cannot load complex into simple in ext/date/date_core.c - Marshal.load(hexdump)

[zdavatz]

after an upgrade from Ruby 2.5 to Ruby 2.7.1 we are confronted with the following problem:

  def test_marshall
    hexdump = "0408553a09446174655b0b690069006902c0a8553a0d526174696f6e616c5b076c2b0800001a71180269029dff6900660c32323939313631"
    ODBA::Marshal.load(hexdump)
    @group.instance_eval('@logs = {Time.local(2011,2,3) => "value"}')
    assert_equal([2011], @group.years)
  end

Looking at the source code of ruby 2.7.1 I find the string cannot load complex into simple in ext/date/date_core.c. Therefore I think it a problem when restoring a date object.

Question: What exactly changed between Ruby 2.5.0 and 2.7.1 in regards to Marshal.load(hexdump)?

There seems to be subtle change in the marshaling/unmarshalling of the date format between Ruby 2.5.0 and 2.7.1

Loading logs with a marshalled dump done with Ruby 2.5.0 will not load with Ruby 2.7.1. ;(

[zdavatz]

Also for reference: https://twitter.com/zdavatz/status/1270997168673996800?s=20

[zdavatz]

interesting commit: 4ffc3953f91ed6d4891853b04d078dad804f1354

[ngiger]

Simple testcase is:

require 'date' hexdump = "0408553a09446174655b0b690069006902c0a8553a0d526174696f6e616c5b076c2b0800001a71180269029dff6900660c32323939313631" binary = [hexdump].pack('H*') obj = Marshal.load(binary) pp obj

Which outputs for Ruby 2.5.0:

<Date: -4712-01-01 ((0j,43200s,(2304000000000/65437)n),+0s,2299161j)>`

and for Ruby 2.7.1

Traceback (most recent call last): 2: from test_date.rb:5:in <main>' 1: from test_date.rb:5:inload' test_date.rb:5:in `marshal_load': cannot load complex into simple (ArgumentError)

[zdavatz]

also see: https://github.com/zdavatz/oddb.org/issues/104

[zdavatz]

Ruby 2.7.1 uses date.gem-3.0.0 as default. Ruby 2.5.0 use no date.gem as default.

[nobu]

That hexdump is invalid.

$ ruby2.5 -rdate -e 'p Marshal.dump(Date.new(-4712,1,1)).unpack("H*")'
["0408553a09446174655b0b69006900690069006900660c32323939313631"]

[zdavatz]

That hexdump is invalid.

$ ruby2.5 -rdate -e 'p Marshal.dump(Date.new(-4712,1,1)).unpack("H*")'
["0408553a09446174655b0b69006900690069006900660c32323939313631"]

Thank you @nobu - but the object can be unmarshalled with Ruby 2.5.0 but not with 2.7.1. Can Ruby 2.5.0 deal better with invalid hexdumps?

[nobu]

As Ruby 2.5.x bundled date 1.0.0 as a default gem, you can install a newer version as a gem.

$ gem2.5 install --user date
Fetching: date-3.0.0.gem (100%)
Building native extensions. This could take a while...
Successfully installed date-3.0.0
Parsing documentation for date-3.0.0
Installing ri documentation for date-3.0.0
Done installing documentation for date after 0 seconds
1 gem installed

$ ruby2.5 --disable=gems -rdate -e 'puts Marshal.load(p ["0408553a09446174655b0b690069006902c0a8553a0d526174696f6e616c5b076c2b0800001a71180269029dff6900660c32323939313631"].pack("H*"))'
"\x04\bU:\tDate[\vi\x00i\x00i\x02\xC0\xA8U:\rRational[\al+\b\x00\x00\x1Aq\x18\x02i\x02\x9D\xFFi\x00f\f2299161"
-4712-01-01

$ ruby2.5 -rdate -e 'puts Marshal.load(p ["0408553a09446174655b0b690069006902c0a8553a0d526174696f6e616c5b076c2b0800001a71180269029dff6900660c32323939313631"].pack("H*"))'
"\x04\bU:\tDate[\vi\x00i\x00i\x02\xC0\xA8U:\rRational[\al+\b\x00\x00\x1Aq\x18\x02i\x02\x9D\xFFi\x00f\f2299161"
Traceback (most recent call last):
    2: from -e:1:in `<main>'
    1: from -e:1:in `load'
-e:1:in `marshal_load': cannot load complex into simple (ArgumentError)
bash: exit 1

[zdavatz]

Yes, but why can't Ruby-2.7.1 deal with the marshaled data of Ruby-2.5.0?

[zdavatz]

or put differently, why can't date-3.0.0.gem deal with marshaled data of date-1.0.0.gem as you show above?

[jeremyevans]

This is a general issue with date starting with date 2.0.0 (ruby 2.6), where certain dates cannot be unmarshalled, even by the same version that marshalled them. This code works on 2.5, but fails on 2.6 and 2.7:

Marshal.load(Marshal.dump(Date.new + 1/2r + 2304/65437r/86400))

[zdavatz]

@jeremyevans looking forward to a new release of date.gem that we can test.

[jeremyevans]

You should probably test the date master branch to make sure it works before we release a gem, after we've fixed the problem. The pull request I submitted has issues, I'll see if I can make changes that allow it to work.

[zdavatz]

@nobu and @hsbt should check your commit. I just started sponsoring @hsbt on github. ;) we will test it too.

[zdavatz]

@jeremyevans I pulled your patch a done and bundle exec rake install with our above test-script I now get:

#<Date: -4712-01-01 ((0j,43200s,(2304000000000/65437)n),+0s,2299161j)>
free(): invalid next size (fast)
Abgebrochen

using ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]

[jeremyevans]

"The pull request I submitted has issues, I'll see if I can make changes that allow it to work."

[jeremyevans]

@zdavatz Pull request #21 should be good now.

[zdavatz]

@jeremyevans awesome! Thank you. Great OpenSource Ruby Spirit. I love it!

[zdavatz]

@jeremyevans yes, it works great:

/tmp> ruby niklaus.rb 
#<Date: -4712-01-01 ((0j,43200s,(2304000000000/65437)n),+0s,2299161j)>

[zdavatz]

@hsbt any chances of getting a date-3.0.1.gem anytime soon? Would be nice for the users.

[hsbt]

https://github.com/ruby/date/pull/21 is not merged yet.

[zdavatz]

👍👍

[zdavatz]

Aligatoo @nobu - great work! Thank you @jeremyevans great work!

[zdavatz]

looking forward to the 3.0.1 release @hsbt