Impossible to sync from genesis

[Nazgolze]

Expected behavior

It should be possible to sync a node from genesis in a timely manner (2-3 days?)

Actual behavior

I'm looking at the logs and it varies, but it usually starts off syncing a block once every 3 seconds and gets progressively slower until it's 1 block every 20-30 seconds (slower than the chain moving)

Steps to reproduce

Sync a node on testnet from genesis

Which version(s) does this affect? (Environment, OS, etc...)

lisk 4.0 up to and including 4.0.0-rc.6

[Nazgolze]

2023-10-11T19:40:48.608Z INFO temp engine 11376 [id=9d8670298eaae7d0895246c956fdb0aa32393eb25cf8e96ce6bd30f30e45d9a4 height=20454723 generator=lsk5ghncy7snfqckqeq4h48bdu98dkxqx5okc27sj numberOfTransactions=0 numberOfAssets=1 numberOfEvents=2] Block executed
2023-10-11T19:40:48.609Z INFO temp engine 11376 [status=success event=chain_newBlock] IPCServer published event
2023-10-11T19:40:48.609Z INFO temp engine 11376 [status=success event=network_newBlock] IPCServer published event
2023-10-11T19:40:48.609Z INFO temp engine 11376 [status=success event=chain_newBlock] WSServer published event
2023-10-11T19:40:48.609Z INFO temp engine 11376 [status=success event=network_newBlock] WSServer published event
2023-10-11T20:33:19.844Z INFO temp engine 11376 [id=7a2973d4abdd340c3adf5426d4387c6c5e4d3ae1e203840eaf0a516965c4c2db height=20454724 generator=lskoxftp2mcvdk7rsdt3vgz66mks7hhs49fta9eoa numberOfTransactions=0 numberOfAssets=1 numberOfEvents=2] Block executed
2023-10-11T20:33:19.845Z INFO temp engine 11376 [status=success event=chain_newBlock] IPCServer published event
2023-10-11T20:33:19.845Z INFO temp engine 11376 [status=success event=network_newBlock] IPCServer published event
2023-10-11T20:33:19.845Z INFO temp engine 11376 [status=success event=chain_newBlock] WSServer published event
2023-10-11T20:33:19.845Z INFO temp engine 11376 [status=success event=network_newBlock] WSServer published event

<--- Last few GCs --->

[11376:0x6b6bea0] 32885010 ms: Scavenge 1903.3 (1981.2) -> 1891.2 (1981.4) MB, 13.2 / 0.0 ms  (average mu = 0.383, current mu = 0.463) allocation failure;
[11376:0x6b6bea0] 32886259 ms: Scavenge (reduce) 1905.0 (1981.4) -> 1892.8 (1953.7) MB, 21.1 / 0.0 ms  (average mu = 0.383, current mu = 0.463) allocation failure;
[11376:0x6b6bea0] 32886435 ms: Scavenge (reduce) 1894.9 (1953.7) -> 1892.9 (1950.9) MB, 11.6 / 0.0 ms  (average mu = 0.383, current mu = 0.463) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7a940 node::Abort() [/home/lisk/lisk-core/bin/node]
 2: 0xa8e823  [/home/lisk/lisk-core/bin/node]
 3: 0xd5c990 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/lisk/lisk-core/bin/node]
 4: 0xd5cd37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/lisk/lisk-core/bin/node]
 5: 0xf3a435  [/home/lisk/lisk-core/bin/node]
 6: 0xf3b338 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/lisk/lisk-core/bin/node]
 7: 0xf4b843  [/home/lisk/lisk-core/bin/node]
 8: 0xf4c6b8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/lisk/lisk-core/bin/node]
 9: 0xf4f885 v8::internal::Heap::HandleGCRequest() [/home/lisk/lisk-core/bin/node]
10: 0xecd9df v8::internal::StackGuard::HandleInterrupts() [/home/lisk/lisk-core/bin/node]
11: 0x12ce115 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/home/lisk/lisk-core/bin/node]
12: 0x16fb6f9  [/home/lisk/lisk-core/bin/node]
lisk-core/bin/lisk-core: line 54: 11376 Aborted                 (core dumped) "$NODE" "$DIR/run" "$@"

[mosmartin]

@Nazgolze I can't seem to replicate this issue. Is it still persistent for you? Also, what are the specs of your node? Based on the error JavaScript heap out of memory looks like a resource exhaustion?

[Nazgolze]

@mosmartin it was a linux machine with 4GB of ram

[mosmartin]

We couldn't reproduce the issue. If the issue crops up, we'll reopen.