search

ruby / openssl

Provides SSL, TLS and general purpose cryptography.
Other
241 stars 163 forks source link

uninitialized constant OpenSSL::KDF #180

Closed Jehops closed 6 years ago

Jehops commented 6 years ago

After upgrading to version 2.1.0, we are seeing the issue described in https://github.com/tootsuite/mastodon/issues/6059. Downgrading back to 2.0.6 fixes the issue.

rhenium commented 6 years ago

From tootsuite/mastodon#6059:

Hi,

jrm (FreeBSD packager) reported on IRC that after updating to Mastodon v2.1.0, Sidekiq and Puma raise the following error:

uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac'
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:19:in `pbkdf2_hmac_sha1'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
/usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
/usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
/usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
/usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
/usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
/usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
/usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
/usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:257:in `boot_system'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:54:in `run'
/usr/local/lib/ruby/gems/2.3/gems/sidekiq-5.0.5/bin/sidekiq:12:in `<top (required)>'
/usr/local/bin/sidekiq:23:in `load'
/usr/local/bin/sidekiq:23:in `<main>'

I believe this is because OpenSSL::KDF was added in Ruby 2.4.

Correction: it's new in openssl gem v2.1.0 which was released a few days ago.

This is still weird because the extension library (openssl.so), which should be loaded by require "openssl", must have defined OpenSSL::KDF module. Could you check what is inside $LOADED_FEATURES?

puts $LOADED_FEATURES.grep(/openssl/)
Jehops commented 6 years ago 
$ irb
irb(main):001:0> puts $LOADED_FEATURES.grep(/openssl/)
=> nil
irb(main):002:0>
Jehops commented 6 years ago

Sorry, I think/hope you meant this.

$ irb
irb(main):001:0> require "openssl"
=> true
irb(main):002:0> puts $LOADED_FEATURES.grep(/openssl/)
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
=> nil
irb(main):003:0>
rhenium commented 6 years ago

Sorry that I wasn't clear enough. Please inject that into the code where the problem exists, say, into /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb, just before line 14.

I suspect the openssl.so from Ruby 2.3 and lib/openssl/pkcs5.rb from openssl gem v2.1.0 are loaded at the same time.

Jehops commented 6 years ago

Here it is the output with the line added.

[71269] * Version 3.11.0 (ruby 2.3.6-p384), codename: Love Song
[71269] * Min threads: 5, max threads: 5
[71269] * Environment: production
[71269] * Process workers: 2
[71269] * Preloading application
/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
[71269] ! Unable to load application: NameError: uninitialized constant OpenSSL::KDF
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:15:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
        from /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:20:in `pbkdf2_hmac_sha1'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:21:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/key_generator.rb:36:in `generate_key'
        from /usr/local/lib/ruby/gems/2.3/gems/globalid-0.4.1/lib/global_id/railtie.rb:26:in `block (2 levels) in <class:Railtie>'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
        from /usr/local/lib/ruby/gems/2.3/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
        from /usr/local/lib/ruby/2.3/tsort.rb:228:in `block in tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:431:in `each_strongly_connected_component_from'
        from /usr/local/lib/ruby/2.3/tsort.rb:349:in `block in each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `call'
        from /usr/local/lib/ruby/2.3/tsort.rb:347:in `each_strongly_connected_component'
        from /usr/local/lib/ruby/2.3/tsort.rb:226:in `tsort_each'
        from /usr/local/lib/ruby/2.3/tsort.rb:205:in `tsort_each'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
        from /usr/local/lib/ruby/gems/2.3/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
        from /usr/local/www/mastodon/config/environment.rb:5:in `<top (required)>'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/lib/ruby/site_ruby/2.3/rubygems/core_ext/kernel_require.rb:55:in `require'
        from config.ru:4:in `block in <main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'
        from config.ru:in `new'
        from config.ru:in `<main>'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `eval'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:49:in `new_from_string'
        from /usr/local/lib/ruby/gems/2.3/gems/rack-2.0.3/lib/rack/builder.rb:40:in `parse_file'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:318:in `load_rackup'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/configuration.rb:243:in `app'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/runner.rb:138:in `load_and_bind'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cluster.rb:397:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/launcher.rb:183:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/lib/puma/cli.rb:77:in `run'
        from /usr/local/lib/ruby/gems/2.3/gems/puma-3.11.0/bin/puma:10:in `<top (required)>'
        from /usr/local/bin/puma:23:in `load'
        from /usr/local/bin/puma:23:in `<main>'
rhenium commented 6 years ago

/usr/local/lib/ruby/2.3/amd64-freebsd11/openssl.so /usr/local/lib/ruby/2.3/openssl/bn.rb /usr/local/lib/ruby/2.3/openssl/pkey.rb /usr/local/lib/ruby/2.3/openssl/cipher.rb /usr/local/lib/ruby/2.3/openssl/config.rb /usr/local/lib/ruby/2.3/openssl/digest.rb /usr/local/lib/ruby/2.3/openssl/x509.rb /usr/local/lib/ruby/2.3/openssl/buffering.rb /usr/local/lib/ruby/2.3/openssl/ssl.rb /usr/local/lib/ruby/2.3/openssl.rb /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb

Well, this is strange -- since (one of the dependencies of) mastodon has openssl as its runtime dependency, only files from the gem should have been loaded. This implies something is doing require "openssl" before Bundler activates the gem (with bundle exec or Bundler.setup), but I I wasn't able to reproduce on Travis CI.

https://travis-ci.org/rhenium/mastodon/jobs/318783772

saper commented 6 years ago

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

saper commented 6 years ago

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

saper commented 6 years ago

To reproduce this problem, try my test_kdf_repr branch of this project.

What is interesting, test_pkcs5.rb from v2.0.6 is fully working with a stock openssl.so

@Jehops we are also running into interesting FreeBSD-related problems additionally. Ruby links to the base system OpenSSL and the port will use one from ports if installed:

/usr/local/bin/ruby24 -I./lib -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.2n  7 Dec 2017
OpenSSL 1.0.2n  7 Dec 2017
Stock Ruby module:
/usr/local/bin/ruby24 -ropenssl -ve'puts OpenSSL::OPENSSL_VERSION, OpenSSL::OPENSSL_LIBRARY_VERSION'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
OpenSSL 1.0.1s-freebsd  1 Mar 2016
OpenSSL 1.0.1u-freebsd  22 Sep 2016
saper commented 6 years ago

Running a whole test suite against the stock openssl.so (not the one from the gem) gives interesting results, including crashes:

I am not sure that having two binary modules of the same name is a good idea and two different require "openssl" APIs. Maybe this gem could detect that required binary symbols are not present and quickly back off?

Full log (except for crash): https://gist.github.com/saper/f96b27e3f2fb14ccf4ada382d8d6a6b7 (running https://github.com/saper/openssl/commit/ed48775f24424e299389c21db3b49ccc515fbce0).

rhenium commented 6 years ago

I have just replaced rubygem-openssl 2.0.6 with rubygem-openssl 2.1.0 in my existing mastodon 2.0.0_3 setup and that was enough to reproduce the issue.

It seems that puma does require 'openssl' and it does not care whether it's this gem or anything else.

Adding gem 'openssl' as the first line there fixes the error, but this is certainly no solution; and also it raises the question why did it work with rubygem-openssl 2.0.6 ? Why does it not complain about undefined OpenSSL::PKCS5 with 2.0.6?

Does adding gem "openssl" to the code work? As mastodon has openssl in its Gemfile, require "openssl" called in Bundler environment should behave as if there were gem "openssl". It has become apparent Bundler has some trouble activating the gem, but I have no idea what's happening -- why don't I reproduce on Travis CI?

I guess you did not see the error with 2.0.6 just because the difference between 2.0.6 and the stdlib one of Ruby 2.4 (= 2.0.x) is so small.

I just noticed that

OpenSSL::PKCS5.pbkdf2_hmac OpenSSL::PKCS5.pbkdf2_hmac_sha1

are defined in the /usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so module shipped with ruby. So rubygem-openssl 2.1.0 tries to override them in Ruby but there is no guarantee that openssl.so provided via the gem is loaded.

This is very messy and is a consequence of a namespace clash between the built-in module and this gem.

As openssl is a "default gem" in Ruby 2.4 onwards, require "openssl" behaves differently depending on how Bundler is used. But, in any case, files from the gem and from the stdlib one will never be mixed. This is not special to openssl, but applies to any other default gem such as json (in Ruby >= 1.9.3). See also [Feature #5481].

Bundler is not used
The latest version of the gem installed is loaded. (The gemspec is activated.)
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. (The gemspec is activated.)
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded. (The gemspec is not activated.)

In previous versions of Ruby, say Ruby 2.3, openssl was not a "default gem". So:

Bundler is not used
The stdlib one is loaded.
Bundle is used, and Gemfile includes openssl
The specified version of the gem is loaded. The stdlib one will be overridden and not loaded.
Bundler is used, but Gemfile does not include openssl
The stdlib one is loaded.
saper commented 6 years ago

@Jehops do you know why we need openssl dependency to be added explicitly? ostatus2 and rails stuff should run on the stock openssl.so module as well...

saper commented 6 years ago

@rhenium

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

Jehops commented 6 years ago

@saper, the FreeBSD ostatus2 port/package matches what ostatus2 gem calls for: https://rubygems.org/gems/ostatus2/versions/2.0.1.

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

saper commented 6 years ago

Thanks, I can see that this was done as a quick fix for openssl.so incompatibility: https://github.com/tootsuite/ostatus2/pull/2

saper commented 6 years ago

Here's what I think happens:

Therefore any subsequent require "openssl" would use /usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb, but /usr/local/lib/ruby/gems/2.3/extensions/amd64-freebsd-10/2.3/openssl-2.1.0/openssl.so will not be used, because openssl.so has already been loaded into the ruby interpreter by puma.

@rhenium Does the above matches your description of how things should work?

rhenium commented 6 years ago

@saper

But, in any case, files from the gem and from the stdlib one will never be mixed.

So how would you explain the behaviour we are seeing? Is this true also when the stdlib binary module has been loaded into interpreter's memory?

That's why I said weird...

@Jehops

It may be worth noting that bundler is not used with the FreeBSD Mastodon package.

OK, this explains why files from Ruby 2.3's stdlib are loaded even though mastodon's Gemfile.lock has openssl. gem "openssl" must be done before require "openssl" happens so that the gem is properly used in Ruby 2.3 as ostatus2 requests. (By the way, you may run into similar troubles with other libraries too unless you can use Bundler, since many stdlibs are "default gem"-ified this year.)

As a separate issue, still I have no idea why only lib/openssl.rb and lib/openssl/pkcs5.rb from the gem.

rhenium commented 6 years ago

@saper Ah, that explains what's going on. Because openssl is not a default gem in Ruby 2.3, RubyGems could not detect the collision. I could minimize that to:

$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/)'
ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/opt/ruby/2.3.6/lib/ruby/2.3.0/x86_64-linux/openssl.so
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/bn.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/pkey.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/cipher.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/config.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/digest.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/x509.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/buffering.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl/ssl.rb
/opt/ruby/2.3.6/lib/ruby/2.3.0/openssl.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/opt/ruby/2.3.6/lib/ruby/gems/2.3.0/gems/openssl-2.1.0/lib/openssl.rb
saper commented 6 years ago

Yes!

On ruby 2.3 with mastodon:

# ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.5p376 (2017-09-14 revision 59905) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.3/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.3/openssl/bn.rb
/usr/local/lib/ruby/2.3/openssl/pkey.rb
/usr/local/lib/ruby/2.3/openssl/cipher.rb
/usr/local/lib/ruby/2.3/openssl/config.rb
/usr/local/lib/ruby/2.3/openssl/digest.rb
/usr/local/lib/ruby/2.3/openssl/x509.rb
/usr/local/lib/ruby/2.3/openssl/buffering.rb
/usr/local/lib/ruby/2.3/openssl/ssl.rb
/usr/local/lib/ruby/2.3/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.3/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
    from -e:1:in `<main>'

On a clean installation of ruby 2.4 with the following packages installed only:

# pkg -j pureruby24 info
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libedit-3.1.20170329_2,1       Command line editor library
libffi-3.2.1_2                 Foreign Function Interface
libunwind-20170113_1           Generic stack unwinding library
libyaml-0.1.6_2                YAML 1.1 parser and emitter written in C
ruby-2.4.3,1                   Object-oriented interpreted scripting language
ruby24-gems-2.6.14             Package management framework for the Ruby language
rubygem-openssl-2.1.0          Ruby gem that wraps around the OpenSSL library

I am getting the same:

root@pureruby24:/ # ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.4.3p205 (2017-12-14 revision 61247) [amd64-freebsd10]
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/local/lib/ruby/2.4/amd64-freebsd10/openssl.so
/usr/local/lib/ruby/2.4/openssl/bn.rb
/usr/local/lib/ruby/2.4/openssl/pkey.rb
/usr/local/lib/ruby/2.4/openssl/cipher.rb
/usr/local/lib/ruby/2.4/openssl/config.rb
/usr/local/lib/ruby/2.4/openssl/digest.rb
/usr/local/lib/ruby/2.4/openssl/x509.rb
/usr/local/lib/ruby/2.4/openssl/buffering.rb
/usr/local/lib/ruby/2.4/openssl/ssl.rb
/usr/local/lib/ruby/2.4/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl.rb
/usr/local/lib/ruby/gems/2.4/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
    from -e:1:in `<main>'

It looks to me that ruby 2.3 and 2.4 behave similarly here....

What is special here: OpenSSL::PKCS5.pbkdf2_hmac is defined as a C function and the openssl gem 2.1.0 overrides it with a ruby method instead.

saper commented 6 years ago

Possibly related: https://github.com/rubygems/rubygems/issues/1507

rhenium commented 6 years ago
  • mastodon requires bundler/setup in config/boot.rb - @Jehops this is where Gemfile.lock gets re-generated btw.

So Bundler is already a runtime dependency of the mastodon port. The quickest and solid fix for the port would be to let puma run in the Bundler environment.

Let me close this issue, as this is not an issue of openssl anymore.

saper commented 6 years ago

@rhenium using Bundler is not a fix for the require problem we have in https://github.com/ruby/openssl/issues/180#issuecomment-354222155 ...

rhenium commented 6 years ago

@saper

It looks to me that ruby 2.3 and 2.4 behave similarly here....

Apparently Ruby from FreeBSD ports have disabled the "default gem" mechanism for some reason I don't know.

https://github.com/freebsd/freebsd-ports/blob/6de157d78b39cbe7986736613b3cd25958934f11/lang/ruby24/files/patch-tool_rbinstall.rb

Possibly related: rubygems/rubygems#1507

It's irrelevant. It's about the filename extension of extension libraries.

@rhenium using Bundler is not a fix for the require problem we have in https://github.com/ruby/openssl/issues/180#issuecomment-354222155 ...

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

saper commented 6 years ago

I think this is not a FreeBSD-only problem.

I have tried this now with Fedora 25, with openssl-2.1.0 installed in $HOME/.gem via gem install openssl.

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:13: warning: method redefined; discarding old pbkdf2_hmac
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:18: warning: method redefined; discarding old pbkdf2_hmac_sha1
/usr/lib64/ruby/openssl.so
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl.rb
/home/saper/.gem/ruby/gems/openssl-2.1.0/lib/openssl/pkcs5.rb:14:in `pbkdf2_hmac': uninitialized constant OpenSSL::KDF (NameError)
    from -e:1:in `<main>'
[saper@fedora0 ~]$ cat /etc/redhat-release 
Fedora release 25 (Twenty Five)
[saper@fedora0 ~]$ rpm -qa | grep ruby
ruby-devel-2.3.4-64.fc25.x86_64
rubygems-2.5.2.1-64.fc25.noarch
ruby-2.3.4-64.fc25.x86_64
rubypick-1.1.1-5.fc24.noarch
rubygem-io-console-0.4.5-64.fc25.x86_64
rubygem-rdoc-4.2.2-3.fc25.noarch
ruby-libs-2.3.4-64.fc25.x86_64
rubygem-bigdecimal-1.2.8-64.fc25.x86_64
rubygem-json-1.8.3.1-64.fc25.x86_64
ruby-irb-2.3.4-64.fc25.noarch
rubygem-psych-2.1.0-64.fc25.x86_64
rubygem-did_you_mean-1.0.0-64.fc25.x86_64

ruby-libs package contains an older version of openssl:

[saper@fedora0 ~]$ rpm -ql ruby-libs | grep ssl
/usr/lib64/ruby/openssl.so
/usr/share/ruby/drb/ssl.rb
/usr/share/ruby/openssl
/usr/share/ruby/openssl.rb
/usr/share/ruby/openssl/bn.rb
/usr/share/ruby/openssl/buffering.rb
/usr/share/ruby/openssl/cipher.rb
/usr/share/ruby/openssl/config.rb
/usr/share/ruby/openssl/digest.rb
/usr/share/ruby/openssl/pkey.rb
/usr/share/ruby/openssl/ssl.rb
/usr/share/ruby/openssl/x509.rb
/usr/share/ruby/webrick/accesslog.rb
/usr/share/ruby/webrick/ssl.rb
[saper@fedora0 ~]$ objdump -T /usr/lib64/ruby/openssl.so |grep pkcs5
0000000000036cd0 g    DF .text  000000000000007b  Base        Init_ossl_pkcs5
[saper@fedora0 ~]$ objdump -T ~/.gem/ruby/extensions/x86_64-linux/2.3.0/openssl-2.1.0/openssl.so  | grep pkcs5 ; echo $?
1
saper commented 6 years ago

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

rhenium commented 6 years ago

@saper 

[saper@fedora0 ~]$ ruby -vwe'require "openssl"; gem "openssl"; require "openssl"; puts $LOADED_FEATURES.grep(/openssl/); OpenSSL::PKCS5.pbkdf2_hmac("a", "b", 1, 20, "sha1")'
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]

I only wanted to show how the mastodon port was going wrongly by a simplified code. In Ruby 2.3 (and also in FreeBSD's Ruby 2.4 port), where openssl is not a "default gem", the first require loads the stdlib one, and then gem activates the gem, and then the second require loads the gem one. This is wrong since it is not possible to load two different versions of a library in Ruby.

I think the problem with openssl 2.1 is that it breaks C ABI by removing OpenSSL::PKCS5 stuff from the C binding file (Init_ossl_pkcs5 is no longer there) and the backward-compatibility works only on the Ruby level

@rhenium Would you be open for a change that restores OpenSSL::PKCS5 in the C binding and drops the pkcs5.rb workaround? I'd propose to remove OpenSSL::PKCS5 altogether in 3.0.

There is no such thing as C level ABI. Our only public interface is the Ruby level one.

As for the "uninitialized constant OpenSSL::KDF" symptom, its cause is that files from two different versions of the openssl library are mixed -- it is a user error. Indeed it is breaking in a weird way, so this:

There might be room for improvement in RubyGems (e.g., detect the collision and make second require fail with an exception, though I don't think it can be implemented easily), but nothing openssl can do here...

RafaelCamarda commented 6 years ago

I had the same issue here: