search

rowland / fb

Firebird Extension Library for Ruby
64 stars 35 forks source link

RangeError: integer overflow on 32-bit Ruby #49

Closed rowland closed 8 years ago

rowland commented 8 years ago

A regression in 0.7.5 (which was meant to be 0.8.0, because it breaks Ruby 1.8.7 compatibility) can cause integer overflows in 32-bit implementations of Ruby. An investigation is in progress.

@vizcay

vizcay commented 8 years ago

@rowland sorry never tested it on 32bit implementations.. something could have slipped away

I've been using my branch with the bigdecimal implementation since September 2014 in production and didn't have a single problem.

vizcay commented 8 years ago

@rowland the integer overflow happens in the test suit? have u been able to reproduce it?

rowland commented 8 years ago

Haven't setup a 32-bit test environment yet, but the error was quite reproducible on my one remaining 32-bit production server.

On Tue, Apr 19, 2016 at 4:05 PM, Pablo Vizcay notifications@github.com wrote:

@rowland https://github.com/rowland the integer overflow happens in the test suit? have u been able to reproduce it?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/rowland/fb/issues/49#issuecomment-212161893

Brent Rowland

rowland commented 8 years ago

Fixing 32-bit compatibility was as simple as appending LL to the min and max constants, but the tests in NumericDataTypesTestCases.rb appear to assume 64-bit architecture, as they fail only under a 32-bit Ruby.

  1) Failure:
test_decimal_18_18_input_range(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:584]:
RangeError expected but nothing was raised.

  2) Failure:
test_decimal_18_18_max(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:540]:
<#<BigDecimal:9c779c4,'0.9223372036 854775807E1',27(36)>> expected but was
<#<BigDecimal:9c769ac,'-0.1371163664 6E-7',18(36)>>.

  3) Failure:
test_decimal_18_18_min(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:550]:
<#<BigDecimal:9c7ad90,'-0.9223372036 854775808E1',27(36)>> expected but was
<#<BigDecimal:9c7a890,'0.1371163664 6E-7',18(36)>>.

  4) Failure:
test_decimal_18_18_rounding(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:566]:
<#<BigDecimal:9c84548,'0.1E-17',9(36)>> expected but was
<#<BigDecimal:9c83f1c,'0.0',9(36)>>.

  5) Failure:
test_decimal_9_0_max(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:266]:
Failed assertion, no message given.

  6) Failure:
test_decimal_9_0_min(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:278]:
Failed assertion, no message given.

  7) Failure:
test_integer_max(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:92]:
Failed assertion, no message given.

  8) Failure:
test_integer_min(NumericDataTypesTestCases) [/home/brent/src/fb/test/NumericDataTypesTestCases.rb:104]:
Failed assertion, no message given.

129 tests, 1011 assertions, 8 failures, 0 errors, 0 skips
rowland commented 8 years ago

Version 0.8.0 restores 32-bit compatibility, except for NumericDataTypesTestCases.