TypeError: incompatible marshal file format (can't be read) format version 4.8 required; 216.25 given

pravi commented 1 year ago

Describe the problem

omniauth-auth0 3.1 test OmniAuth::Strategies::Auth0 oauth stores session['authorize_params'] as a plain Ruby Hash is failing with following error.

$ bundle exec rake
/usr/bin/ruby3.1 -I/usr/share/rubygems-integration/all/gems/rspec-core-3.12.0/lib:/usr/share/rubygems-integration/all/gems/rspec-support-3.12.0/lib /usr/share/rubygems-integration/all/gems/rspec-core-3.12.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
...................................................................F..............

Failures:

  1) OmniAuth::Strategies::Auth0 oauth stores session['authorize_params'] as a plain Ruby Hash
     Failure/Error: Marshal.load(decoded_session_data)

     TypeError:
       incompatible marshal file format (can't be read)
        format version 4.8 required; 216.25 given
     # ./spec/omniauth/strategies/auth0_spec.rb:206:in `session'
     # ./spec/omniauth/strategies/auth0_spec.rb:212:in `block (3 levels) in <top (required)>'
     # /usr/share/rubygems-integration/all/gems/webmock-3.18.1/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

Finished in 32.78 seconds (files took 1.48 seconds to load)
82 examples, 1 failure

Failed examples:

rspec ./spec/omniauth/strategies/auth0_spec.rb:209 # OmniAuth::Strategies::Auth0 oauth stores session['authorize_params'] as a plain Ruby Hash

/usr/bin/ruby3.1 -I/usr/share/rubygems-integration/all/gems/rspec-core-3.12.0/lib:/usr/share/rubygems-integration/all/gems/rspec-support-3.12.0/lib /usr/share/rubygems-integration/all/gems/rspec-core-3.12.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed

What was the expected behavior?

The test should pass

Reproduction

This was happening the debian unstable with the simplified Gemfile and Gemfile.lock (all using native debian packages)

source 'https://rubygems.org'

gemspec

gem 'jwt', '~> 2'
gem 'rake', '~> 13'
gem 'rexml', '~> 3.2'
gem 'sinatra', '~> 3.0'

group :test do
  gem 'rack-test', '~> 2'
  gem 'rspec', '~> 3'
  gem 'webmock', '~> 3'
  gem 'multi_json', '~> 1'

PATH
  remote: .
  specs:
    omniauth-auth0 (3.1.0)
      omniauth (~> 2)
      omniauth-oauth2 (~> 1)

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.8.1)
      public_suffix (>= 2.0.2, < 6.0)
    crack (0.4.4)
    diff-lcs (1.5.0)
    faraday (1.1.0)
      multipart-post (>= 1.2, < 3)
      ruby2_keywords
    hashdiff (1.0.1)
    hashie (5.0.0)
    jwt (2.5.0)
    multi_json (1.14.1)
    multi_xml (0.6.0)
    multipart-post (2.0.0)
    mustermann (3.0.0)
      ruby2_keywords (~> 0.0.1)
    oauth2 (1.4.4)
      faraday (>= 0.8, < 2.0)
      jwt (>= 1.0, < 3.0)
      multi_json (~> 1.3)
      multi_xml (~> 0.5)
      rack (>= 1.2, < 3)
    omniauth (2.1.1)
      hashie (>= 3.4.6)
      rack (>= 2.2.3)
      rack-protection
    omniauth-oauth2 (1.8.0)
      oauth2 (>= 1.4, < 3)
      omniauth (~> 2.0)
    public_suffix (4.0.6)
    rack (2.2.4)
    rack-protection (3.0.5)
      rack
    rack-test (2.0.2)
      rack (>= 1.3)
    rake (13.0.6)
    rexml (3.2.5)
    rspec (3.12.0)
      rspec-core (~> 3.12.0)
      rspec-expectations (~> 3.12.0)
      rspec-mocks (~> 3.12.0)
    rspec-core (3.12.0)
      rspec-support (~> 3.12.0)
    rspec-expectations (3.12.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.12.0)
    rspec-mocks (3.12.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.12.0)
    rspec-support (3.12.0)
    ruby2_keywords (0.0.5)
    sinatra (3.0.5)
      mustermann (~> 3.0)
      rack (~> 2.2, >= 2.2.4)
      rack-protection (= 3.0.5)
      tilt (~> 2.0)
    tilt (2.0.11)
    webmock (3.18.1)
      addressable (>= 2.8.0)
      crack (>= 0.3.2)
      hashdiff (>= 0.4.0, < 2.0.0)

PLATFORMS
  x86_64-linux

DEPENDENCIES
  bundler
  jwt (~> 2)
  multi_json (~> 1)
  omniauth-auth0!
  rack-test (~> 2)
  rake (~> 13)
  rexml (~> 3.2)
  rspec (~> 3)
  sinatra (~> 3.0)
  webmock (~> 3)

BUNDLED WITH
   2.3.15

Where possible, please include:

The smallest possible sample app that reproduces the undesirable behavior

Log files (redact/remove sensitive information)

Application settings (redact/remove sensitive information)

Screenshots -->

Step 1..
Step 2..
...

Environment

Debian unstable (I'm part of the team that maintained ruby-omniauth-auth0 in debian and I'm trying to update the package to version 3.1.0). This may be due to a different version used in Debian, but I'm not able to pin point which version.

Version of this library used:3.1.0
Which framework are you using, if applicable:
Other modules/plugins/libraries that might be involved: ruby 3.1
Any other relevant information you think would be useful: tried with oauth 2.0.7 but same failure

pravi commented 1 year ago

Also removed simplecov

diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
index c014c12..308c88a 100644
--- a/spec/spec_helper.rb
+++ b/spec/spec_helper.rb
@@ -2,13 +2,6 @@ $LOAD_PATH.unshift File.expand_path(__dir__)
 $LOAD_PATH.unshift File.expand_path('../lib', __dir__)

 require 'multi_json'
-require 'simplecov'
-SimpleCov.start
-
-if ENV['CI'] == 'true'
-  require 'simplecov-cobertura'
-  SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter
-end

 require 'rspec'

stevehobbsdev commented 1 year ago

Thanks for your patience here.

Also removed simplecov

Did this resolve the issue?

The tests in general seem ok, as you say this is likely environmental. They pass on MacOS 12.6.3, I have also successfully run them using this Docker image (mcr.microsoft.com/devcontainers/ruby:3.1).

The Debian specifics are a little out of my wheelhouse but happy to take suggestions on a fix if there is still an issue.

pravi commented 1 year ago

What version of openssl are you using? In debian sid we have openssl 3.

pravi commented 1 year ago

It still failing for me. May be you can try with a debian sid docker image? https://hub.docker.com/_/debian has sid.

stevehobbsdev commented 1 year ago

That appears to be ok. This is what I did:

Pulled debian:sid
apt install ruby make git ruby-dev gcc g++
bundle install
bundle exec rake test

Output:

Fetching gem metadata from https://rubygems.org/.........
Using rake 13.0.6
Using public_suffix 5.0.1
Using ast 2.4.2
Using bundler 2.3.26
Using coderay 1.1.3
Using daemons 1.4.1
Using diff-lcs 1.5.0
Using docile 1.4.0
Using dotenv 2.8.1
Using faraday-net_http 3.0.2
Using ruby2_keywords 0.0.5
Using ffi 1.15.5
Using rb-fsevent 0.11.2
Using lumberjack 1.2.8
Using nenv 0.3.0
Using shellany 0.0.1
Using method_source 1.0.0
Using thor 1.2.1
Using guard-compat 1.2.1
Using rspec-support 3.12.0
Using hashdiff 1.0.1
Using hashie 5.0.0
Using json 2.6.3
Using jwt 2.7.0
Using multi_json 1.15.0
Using multi_xml 0.6.0
Using rack 2.2.6.2
Using version_gem 1.1.1
Using parallel 1.22.1
Using rainbow 3.1.1
Using regexp_parser 2.7.0
Using ruby-progressbar 1.11.0
Using unicode-display_width 2.4.2
Using simplecov-html 0.12.3
Using simplecov_json_formatter 0.1.4
Using tilt 2.0.11
Using gem-release 2.2.2
Using parser 3.2.1.0
Using faraday 2.7.4
Using addressable 2.8.1
Using notiffany 0.1.3
Using pry 0.14.2
Using rspec-core 3.12.1
Using rspec-expectations 3.12.2
Using rspec-mocks 3.12.3
Using mustermann 2.0.2
Using snaky_hash 2.0.1
Using rack-protection 2.2.4
Using rack-test 2.0.2
Using shotgun 0.9.2
Using simplecov 0.22.0
Using rubocop-ast 1.26.0
Using formatador 1.1.0
Using rexml 3.2.5
Fetching rb-inotify 0.10.1
Using crack 0.4.5
Using rspec 3.12.0
Using oauth2 2.0.9
Using simplecov-cobertura 2.1.0
Using sinatra 2.2.4
Using webmock 3.18.1
Using omniauth 2.1.1
Using rubocop 1.45.1
Using omniauth-oauth2 1.8.0
Using omniauth-auth0 3.1.0 from source at `.`
Installing eventmachine 1.2.7 with native extensions
Installing rb-inotify 0.10.1
Fetching listen 3.8.0
Installing listen 3.8.0
Fetching guard 2.18.0
Installing guard 2.18.0
Fetching guard-rspec 4.7.3
Installing guard-rspec 4.7.3
Fetching thin 1.8.1
Installing thin 1.8.1 with native extensions
Bundle complete! 18 Gemfile dependencies, 70 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
root@d566d4f2f887:/app# bundle exec rake test

All examples were filtered out; ignoring {:focus=>true}
....

Finished in 4.93 seconds (files took 1.7 seconds to load)
82 examples, 0 failures

Coverage report generated for RSpec to /app/coverage. 819 / 825 LOC (99.27%) covered.
root@d566d4f2f887:/app# openssl version
OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)
root@d566d4f2f887:/app# ruby -v
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux-gnu]
root@d566d4f2f887:/app#

Let me know if I can help diagnose further.

terceiro commented 1 year ago

this is caused by Debian being at sinatra 3. This can be reproduced with bundler with this change:

diff --git a/Gemfile b/Gemfile
index c6a5572..62a3aba 100644
--- a/Gemfile
+++ b/Gemfile
@@ -11,7 +11,7 @@ group :development do
   gem 'pry', '~> 0'
   gem 'rubocop', '~> 1', require: false
   gem 'shotgun', '~> 0'
-  gem 'sinatra', '~> 2'
+  gem 'sinatra', '~> 3'
   gem 'thin', '~> 1'
 end

Then bundle update (or remove Gemfile.lock and run bundle install again), and the tests will fail exactly like reported:

$ bundle exec rake test
/usr/lib/ruby-standalone/bin/ruby -I/home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-core-3.12.1/lib:/home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-support-3.12.0/lib /home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-core-3.12.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}
...................................................................F......OAuth2::AccessToken.from_hash: `hash` contained more than one 'token' key (["access_token", "id_token"]); using "access_token".
.OAuth2::AccessToken.from_hash: `hash` contained more than one 'token' key (["access_token", "id_token"]); using "access_token".
.OAuth2::AccessToken.from_hash: `hash` contained more than one 'token' key (["access_token", "id_token"]); using "access_token".
.OAuth2::AccessToken.from_hash: `hash` contained more than one 'token' key (["access_token", "id_token"]); using "access_token".
.OAuth2::AccessToken.from_hash: `hash` contained more than one 'token' key (["access_token", "id_token"]); using "access_token".
....

Failures:

  1) OmniAuth::Strategies::Auth0 oauth stores session['authorize_params'] as a plain Ruby Hash
     Failure/Error: Marshal.load(decoded_session_data)

     TypeError:
       incompatible marshal file format (can't be read)
        format version 4.8 required; 205.35 given
     # ./spec/omniauth/strategies/auth0_spec.rb:206:in `session'
     # ./spec/omniauth/strategies/auth0_spec.rb:212:in `block (3 levels) in <top (required)>'
     # /home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/webmock-3.18.1/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

Finished in 6.68 seconds (files took 0.44659 seconds to load)
82 examples, 1 failure

Failed examples:

rspec ./spec/omniauth/strategies/auth0_spec.rb:209 # OmniAuth::Strategies::Auth0 oauth stores session['authorize_params'] as a plain Ruby Hash

Coverage report generated for RSpec to /home/terceiro/src/debian/ruby-team/ruby-omniauth-auth0/coverage. 819 / 825 LOC (99.27%) covered.
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
/usr/lib/ruby-standalone/bin/ruby -I/home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-core-3.12.1/lib:/home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-support-3.12.0/lib /home/terceiro/.ruby-standalone/gems/ruby/3.1.0/gems/rspec-core-3.12.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed

stevehobbsdev commented 1 year ago

Thanks for the insight here @terceiro - I can indeed reproduce the issue after update to sinatra@3.

I'd been keen to update this but I'm not sure I understand why the error is occurring. Happy to hear suggestions on how to resolve it.

In the meantime, possible to use sinatra@2 in your case @pravi?

pravi commented 1 year ago

In Debian, we try to use only a single version of a software as much as possible, only in rare cases (for example moving to a single version takes years), we keep multiple versions. There is only a single version of ruby at any time in one release, but python team kept python 2 and 3 in last release. For sinatra, keeping a separate version for running tests of omniauth-auth0 does not make much sense. In the worse case, if we can't fix it, we might disable that test, now that we know the failure is caused by sinatra update and a bug in omniauth-auth0.

stevehobbsdev commented 1 year ago

So after some hefty debugging trying to figure out what's going on, it turns out the default session store for Sinatra 3 was changed to Rack::Protection::EncryptedCookie, and this code that we have for reading that cookie in the test obviously won't cut it, as it's quite naive.

I did try to use Rack::Protection::Encryptor::decrypt_message in the test as a replacement, but it fails here in the call to cipher.final, and I hit a bit of a wall.

Anyway, I can fix this by using the Sinatra 2 default here of Rack::Session::Cookie as it's only for testing purposes - I'll raise a PR 👍🏻

pravi commented 1 year ago

@stevehobbsdev thanks for the fix. I have included the patch in the debian package and all tests are passing.

Update: build log https://buildd.debian.org/status/fetch.php?pkg=ruby-omniauth-auth0&arch=all&ver=3.1.0-1&stamp=1677670702&raw=0

stevehobbsdev commented 1 year ago

Great!

auth0 / omniauth-auth0