Support Unicode Gem path

davispuh commented 4 weeks ago

I'm using RVM and my $HOME is located at /home/Dāvis which means GEM path is at /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/

Currently this causes bootsnap to fail with

/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `start_with?': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `block in stability'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `detect'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `stability'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:12:in `stable?'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:60:in `entries_and_dirs'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:155:in `block (2 levels) in push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:147:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:147:in `block in push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:53:in `block in transaction'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:52:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:52:in `transaction'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:146:in `push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `block in push_paths'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `push_paths'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/change_observer.rb:11:in `<<'

This is because Gem.path is at ["/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0"] and it's UTF-8 encoded while sometimes path can be with ASCII-8BIT encoding.

Note that there is similar issue within Ruby aswell, see ruby/ruby#11958

casperisfine commented 4 weeks ago

Could you provide some reproducible test case? I'm trying to reproduce your problem but even by creating some mix encoding paths, I'm not able to:

>> Dir["**/*.rb"]
=> ["t€st/\xC3.rb"]
>> Dir["**/*.rb"].map(&:encoding)
=> [#<Encoding:UTF-8>]

I suspect there's more to this problem, like a LOCALE issue perhaps.

davispuh commented 4 weeks ago

I'm not exactly sure what causes to use ASCII-8BIT encoding but this example shows how require path is in different encoding

# encoding: ISO-8859-1

module Kernel
    alias_method :actual_require, :require

    def require(path)
       puts "requiring #{path} with #{path.encoding} encoding"
       actual_require(path)
    end
end

require 'tzinfo/data'

requiring tzinfo/data with ISO-8859-1 encoding
requiring tzinfo/data/version with UTF-8 encoding

My LANG env var is lv_LV.UTF-8 and I don't use any other locale than UTF-8.

casperisfine commented 4 weeks ago

I'm not exactly sure what causes to use ASCII-8BIT encoding

Can you provide the entire backtrace then?

davispuh commented 4 weeks ago

I'm not exactly sure what causes to use ASCII-8BIT encoding

Can you provide the entire backtrace then?

That is whole backtrace, for some reason it doesn't show anything more.

Basically this happens when I try to use Metasploit Framework msfconsole (https://github.com/rapid7/metasploit-framework) but first issue I encounter is Ruby's https://github.com/ruby/ruby/pull/11958 after fixing that I get this one.

Also note that most of requires work fine, only some deeply in loading when it tries to load tzinfo/data it fails.

Here's full backtrace without Ruby's fix

/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:102:in `delete_prefix': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:102:in `warning?'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:71:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/kernel.rb:34:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:149:in `create_default_data_source'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:55:in `block in get'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:54:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:54:in `get'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/activesupport-7.0.8.4/lib/active_support/railtie.rb:88:in `block in <class:Railtie>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:32:in `instance_exec'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:32:in `run'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:61:in `block in run_initializers'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:231:in `block in tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:353:in `block (2 levels) in each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:434:in `each_strongly_connected_component_from'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:352:in `block in each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `call'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:229:in `tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:208:in `tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:60:in `run_initializers'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/application.rb:372:in `initialize!'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/railtie.rb:226:in `public_send'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/railtie.rb:226:in `method_missing'
        from /opt/metasploit/config/environment.rb:4:in `<top (required)>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /opt/metasploit/lib/msfenv.rb:28:in `<top (required)>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /opt/metasploit/msfconsole:21:in `<main>'

casperisfine commented 3 weeks ago

I'm gonna wait to see how https://github.com/rubygems/rubygems/pull/8196 and https://github.com/ruby/ruby/pull/11958 are resolved.

Shopify / bootsnap

Support Unicode Gem path #493