search

iron-fish / ironfish

A novel cryptocurrency focused on privacy and accessibility.
https://ironfish.network
Mozilla Public License 2.0
965 stars 574 forks source link

JavaScript heap out of memory #843

Closed gddrx closed 2 years ago

gddrx commented 2 years ago 
1624985 H/sMining block 44411 on request 49761... 1265503 H/sMining block 44411 on request 49761... 1238290 H/sMining block 44411 on request 49761... 1630517 H/sMining block 44411 on request 49761... 1258933 H/sMining block 44411 on request 49761... 1248536 H/sMining block 44411 on request 49761... 1622018 H/sMining block 44411 on request 49761... 1286179 H/sMining block 44411 on request 49761... 1252179 H/sMining block 44411 on request 49761... 1609843 H/sMining block 44411 on request 49761... 1248020 H/sMining block 44411 on request 49761... 1232020 H/sMining block 44411 on request 49761... 1611858 H/sMining block 44411 on request 49761... 1251849 H/sMining block 44411 on request 49761... 1241388 H/sMining block 44411 on request 49761... 1575609 H/s
Dec 30 07:39:16 dell00 yarn[3305883]: <--- Last few GCs --->
Dec 30 07:39:16 dell00 yarn[3305883]: [3305883:0x4e3c1a0] 109014798 ms: Mark-sweep 3848.2 (4129.9) -> 3834.9 (4130.1) MB, 4693.4 / 0.0 ms  (average mu = 0.335, current mu = 0.323) task scavenge might not succeed
Dec 30 07:39:16 dell00 yarn[3305883]: [3305883:0x4e3c1a0] 109021501 ms: Mark-sweep 3848.1 (4130.1) -> 3835.1 (4129.9) MB, 4623.4 / 0.0 ms  (average mu = 0.323, current mu = 0.310) task scavenge might not succeed
Dec 30 07:39:16 dell00 yarn[3305883]: <--- JS stacktrace --->
Dec 30 07:39:16 dell00 yarn[3305883]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Dec 30 07:39:16 dell00 yarn[3305883]:  1: 0xb00d90 node::Abort() [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  2: 0xa1823b node::FatalError(char const*, char const*) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  3: 0xcedbce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  4: 0xcedf47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  5: 0xea6105  [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  6: 0xea6be6  [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  7: 0xeb4b1e  [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  8: 0xeb5560 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]:  9: 0xf18aa4 v8::internal::ScavengeJob::Task::RunInternal() [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 10: 0xdabeeb non-virtual thunk to v8::internal::CancelableTask::Run() [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 11: 0xb6e964 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 12: 0xb707c9 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 13: 0x155a336  [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 14: 0x156c7a4  [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 15: 0x155ac68 uv_run [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 16: 0xa3dfe5 node::SpinEventLoop(node::Environment*) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 17: 0xb42dc6 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 18: 0xac4812 node::Start(int, char**) [/usr/bin/node]
Dec 30 07:39:16 dell00 yarn[3305883]: 19: 0x7f8cd9a760b3 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
Dec 30 07:39:16 dell00 yarn[3305883]: 20: 0xa3bf3c  [/usr/bin/node]
Dec 30 07:41:35 dell00 yarn[3305862]: error Command failed with exit code 1.
Dec 30 07:41:35 dell00 yarn[3305862]: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

v 0.1.15 This miner connected to my first server with node. Nothing do for repeat, but srv have 64Gb RAM+ 128Gb SWAP

Shekelme commented 2 years ago

Some lines from my recent crashes:

SEND WebRtc DTFiU+k: NewTransaction (8) (repeated 2 times)
SEND WebSocket C2APnTc: NewTransaction (8) (repeated 2 times)
RECV WebRtc Hbp0tUr: NewTransaction (8) (repeated 70 times)
RECV WebRtc nfB76qB: NewTransaction (8) (repeated 55 times)
RECV WebRtc y0tWbtM: NewTransaction (8) (repeated 49 times)
RECV WebRtc OoVkhc9: NewTransaction (8) (repeated 46 times)
RECV WebRtc 8Rcs+tJ: NewTransaction (8) (repeated 70 times)
RECV WebRtc qk+TVjA: NewTransaction (8) (repeated 29 times)
RECV WebRtc Z8DxyOi: NewTransaction (8) (repeated 24 times)
RECV WebRtc xIPucuC: NewTransaction (8) (repeated 45 times)

<--- Last few GCs --->

[1643332:0x56e46c0] 62093929 ms: Mark-sweep 4017.2 (4134.2) -> 4002.1 (4134.5) MB, 2342.0 / 0.1 ms  (average mu = 0.301, current mu = 0.255) allocation failure scavenge might not succeed
[1643332:0x56e46c0] 62097129 ms: Mark-sweep 4018.1 (4134.5) -> 4002.6 (4135.0) MB, 2083.6 / 0.2 ms  (average mu = 0.325, current mu = 0.349) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09c10 node::Abort() [node]
 2: 0xa1c193 node::FatalError(char const*, char const*) [node]
 3: 0xcf8dbe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcf9137 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xeb09d5  [node]
 6: 0xeb14b6  [node]
 7: 0xebf9de  [node]
 8: 0xec0420 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xec339e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xe848da v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x11fd626 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x15f2099  [node]

NPM installation method, Node.js v16.15.0, Ubuntu server 20.04.4 LTS, AMD Ryzen Threadripper PRO 3955WX 16-Cores, 512GB RAM, 931.5G swap partition. :-|

Shekelme commented 2 years ago

v0.1.39, nothing has changed, still the same error.

lwisne commented 2 years ago

What is the behavior before you see the error? Does it run at all? Or does it take many hours/days before you see the error?

Shekelme commented 2 years ago

The node just works. But it is almost guaranteed to crash with such an error if you give it a day or two to work. Usually I constantly monitor it's status with ironfish status -f command.

lwisne commented 2 years ago

OK - we are currently working on memory management for long running cases like this. Should see some updates soon!

Shekelme commented 2 years ago

Please note that I have extended memory limit with export NODE_OPTIONS=--max_old_space_size=32768 and I'm still getting this error.

lwisne commented 2 years ago

Duplicate of https://github.com/iron-fish/ironfish/issues/1925