FATAL ERROR from version 1.6.37 onwards

[dadepo]

npm version: 6.13.4

I am trying to upgrade from v1.6.28 to 1.6.48 and when I run the test for the project, I get the following errors:

<--- Last few GCs --->

[54710:0x103ee5000]  1673117 ms: Mark-sweep 2046.3 (2050.7) -> 2045.3 (2050.7) MB, 6352.3 / 0.0 ms  (average mu = 0.100, current mu = 0.000) allocation failure scavenge might not succeed
[54710:0x103ee5000]  1677135 ms: Mark-sweep 2046.3 (2050.7) -> 2045.3 (2050.7) MB, 4012.9 / 0.0 ms  (average mu = 0.063, current mu = 0.001) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x10074d059]
Security context: 0x350001cc0921 <JSObject>
    1: divModAny(aka divModAny) [0x3500c07c0ae9] [/Users/me/Desktop/project/ip-num/node_modules/big-integer/BigInteger.js:~544] [pc=0x3a8e5f10aa3f](this=0x3500784804b9 <undefined>,0x35006fb41349 <NativeBigInt map = 0x3500ac4c1071>,0x35006fb41259 <NativeBigInt map = 0x3500ac4c1071>)
    2: toBase(aka toBase) [0x3500c07c0e49] [/Users/me...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191228.000804.54710.0.001.json
Node.js report completed
 1: 0x100b9d1be node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x100083888 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1000839b0 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x10017cc41 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10017cbeb v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x100296ba3 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x100297f28 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
 8: 0x100295af6 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x1002945d5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x10029bf74 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x10029bfca v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x100279ccf v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
13: 0x1004dea6b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x10074d059 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
^C^C
<--- Last few GCs --->

[54736:0x1024a1000]  1857719 ms: Scavenge 2046.9 (2050.5) -> 2046.0 (2050.5) MB, 7.9 / 0.0 ms  (average mu = 0.371, current mu = 0.468) allocation failure 
[54736:0x1024a1000]  1857733 ms: Scavenge 2046.9 (2050.5) -> 2046.0 (2050.5) MB, 11.7 / 0.0 ms  (average mu = 0.371, current mu = 0.468) allocation failure 
[54736:0x1024a1000]  1858074 ms: Scavenge 2046.9 (2050.5) -> 2046.0 (2050.8) MB, 91.1 / 0.0 ms  (average mu = 0.371, current mu = 0.468) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x10074d059]
Security context: 0x0f4e29a00921 <JSObject>
    1: divModAny(aka divModAny) [0xf4eb6e40ca1] [/Users/me/Desktop/project/ip-num/node_modules/big-integer/BigInteger.js:~544] [pc=0x328ec29a72e0](this=0x0f4ecf2404b9 <undefined>,0x0f4ecb80d419 <NativeBigInt map = 0xf4e939010c1>,0x0f4ecb80ada1 <NativeBigInt map = 0xf4e939010c1>)
    2: toBase(aka toBase) [0xf4e9ffef3c9] [/Users/me...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191228.001837.54736.0.001.json
Node.js report completed
 1: 0x100b9d1be node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x100083888 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1000839b0 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x10017cc41 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10017cbeb v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x100296ba3 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x100297f28 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
 8: 0x100295af6 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x1002945d5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x10029bf74 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x10029bfca v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x100279ccf v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
13: 0x1004dea6b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x10074d059 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]

I was able to narrow this down to version 1.6.37 in the sense that version 1.6.36 and below runs fine, but from version 1.6.37 and above, it does not.

[peterolson]

Version 1.6.38 was released 1 year ago in December 2018.

It seems that the code you are running was close to reaching the memory limit, and some small change in the library pushed it over the limit.

Could you share the code you are running?

[dadepo]

So I was able to narrow down the part of the test code that triggers this behaviour. It can be found here https://github.com/ip-num/ip-num/blob/69088bfa38f7a84ffad7b38d4da66b1859bbf3a2/spec/IPv6CidrRangeTest.ts#L51

In summary this is what the test is doing:

1. Construct an IPv6 with a certain size
2. Get that size and add one to it, to get a retrieve size
3. Attempt to take the retrieve size out of the IPV6
4. Assert that an error is thrown because retrieve size is bigger (by one) than the size of IPv6

The size being used for this test is (2 ^ 82)

Observation using the previous (< 1.6.37) version of big-integer:

IPV6.getSize().valueOf() prints 4.835703278458517e+24 // js number IPV6.getSize().plus(1).valueOf() prints 4.835703278458517e+24 // js number

and

IPV6.getSize() prints:

 Integer {
  value: [ 0, 5851700, 7032784, 4835 ],
  sign: false,
  isSmall: false
}

and

IPV6.getSize().plus(1) prints:

Integer {
  value: [ 8824704, 5851669, 7032784, 4835 ],
  sign: false,
  isSmall: false
}

and

IPV6.getSize().plus(1).greater(IPV6.getSize()) returns true, which is expected

Observation using the previous (>= 1.6.37) version of big-integer:

IPV6.getSize().valueOf() prints 4.835703278458517e+24 // js number IPV6.getSize().plus(1).valueOf() prints 4.835703278458517e+24 // js number

and

IPV6.getSize() prints:

Integer { value: 4835703278458516698824704n }

and

IPV6.getSize().plus(1) prints:

Integer { value: 4835703278458516698824704n }

and

IPV6.getSize().plus(1).greater(IPV6.getSize()) now returns false, which is not desired, which then lead the code to fill an array up with (2 ^ 82) instances of an object, and hence the crash.

So I guess the question is, why has the .greater change? is this something I am doing wrong?

[Yaffle]

Why does IPV6.getSize().plus(1) returns the same value in your test?

[dadepo]

Why does IPV6.getSize().plus(1) returns the same value in your test?

Exactly the same question I asked myself...

But I noticed that, even though same value are also printed in versions < 1.6.37, the greater than checks behaves as I would expect it.

[Yaffle]

@dadepo ,

But I noticed that, even though same value are also printed in versions < 1.6.37, the greater than checks behaves as I would expect it.

You are saying, that it is valueOf returns the same values, but this is because it is converted to the number here. And you are saying, that the arrays in <1.6.37 are different and in the new version the internal value is the same, which is really strange to me. I would try to debug your code trying to check if 4835703278458516698824704n+1n works correctly, then trying to double check again the test and by trying to set a break point before the check.

I cannot reproduce with some simple test. And I don't know how to run the tests in your lib.

[dadepo]

I cannot reproduce with some simple test. And I don't know how to run the tests in your lib.

I pushed the branch I used for poking around. https://github.com/ip-num/ip-num/tree/debugging-oom-error it has all the console logs in both the test and the implementation.

If you can out the branch and run npm run test, you should be able to see the output. (you need to have TypeSript installed though).

Only one test will run. You can the change the big-integer version in package.json, rerun the test and see the output also.

Hopefully this helps...

[dadepo]

I was able to isolate the reproduction steps. If you create a node project and run the code below in the terminal, you will see the behaviour I encountered reproduced.

var bigInt = require("big-integer");
let v1 = bigInt(2).pow(82);
let v1_plus_one = v1.plus(1)

console.log("To string representation of valueOf:")
console.log(v1.valueOf())
console.log(v1_plus_one.valueOf())

// making an instance of bigInt again from v1 after adding 1 to it
v1_plus_one2 = bigInt(v1_plus_one.valueOf())

// Expect this to print true, but it prints false
console.log(v1_plus_one2.greater(v1))                                      

my guess is conversion to number (via valueOf) and then constructing the bigInt again leads to loss in precision? Would appreciate if you can help confirm that this is what I am doing wrong and I can change my code not to.

UPDATE

It seems this is the case. console.log(v1_plus_one.greater(v1)) prints true as expected. Once I get confirmation from your end regarding this behaviour and it not being an issue... I shall close the ticket. But it would be interesting to know why this never caused any problems in previous versions.

[Yaffle]

Yes, valueOf does conversion to number with precision loss.

[dadepo]

Any thoughts on why this never caused any problems in previous versions?

[Yaffle]

it is hard to say - I cannot find it from the code

[dadepo]

Maybe it might be interesting to know that, this changed in behaviour started with the 1.6.37 release, which was the release that added the ability for the library to act as a polyfill to native BigInt https://github.com/peterolson/BigInteger.js/releases/tag/v1.6.37