graphprotocol / indexer

Graph Protocol indexer components and infrastructure
MIT License
238 stars 127 forks source link

indexer-agent crash #552

Open grumas-rc opened 1 year ago

grumas-rc commented 1 year ago

<--- Last few GCs --->

[66493:0x5b962c0] 897845529 ms: Scavenge 3985.3 (4140.6) -> 3984.1 (4140.6) MB, 4.2 / 0.0 ms (average mu = 0.999, current mu = 0.999) allocation failure [66493:0x5b962c0] 897845535 ms: Scavenge 3985.8 (4140.6) -> 3984.5 (4140.6) MB, 2.9 / 0.0 ms (average mu = 0.999, current mu = 0.999) allocation failure [66493:0x5b962c0] 897846679 ms: Scavenge 3986.4 (4140.8) -> 3984.8 (4144.8) MB, 8.7 / 0.0 ms (average mu = 0.999, current mu = 0.999) allocation failure

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0xb06730 node::Abort() [node] 2: 0xa1b6d0 [node] 3: 0xce1dd0 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xce2177 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xe997e5 [node] 6: 0xea94ad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 7: 0xeac1ae v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node] 8: 0xe6d6ea v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node] 9: 0x11e6456 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long, v8::internal::Isolate) [node]

graph-indexer-agent --version 0.20.5-alpha.1

node -v v16.18.0

Ubuntu 20.04.5 LTS

memory usage is growing all the time indexer-agent

grumas-rc commented 1 year ago

Also happened on the 0.20.8 version

<--- Last few GCs --->

[6181:0x554d260] 897616011 ms: Scavenge 3973.5 (4129.3) -> 3967.8 (4129.3) MB, 9.6 / 0.0 ms (average mu = 0.997, current mu = 0.996) task [6181:0x554d260] 897620565 ms: Scavenge 3975.4 (4130.0) -> 3969.2 (4130.5) MB, 10.0 / 0.0 ms (average mu = 0.997, current mu = 0.996) allocation failure [6181:0x554d260] 897626232 ms: Scavenge 3975.4 (4130.5) -> 3969.8 (4147.0) MB, 10.0 / 0.0 ms (average mu = 0.997, current mu = 0.996) task

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0xb08e80 node::Abort() [node] 2: 0xa1b70e [node] 3: 0xce1890 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xce1c37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xe992a5 [node] 6: 0xea8f6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 7: 0xf0c194 v8::internal::ScavengeJob::Task::RunInternal() [node] 8: 0xd9f84b non-virtual thunk to v8::internal::CancelableTask::Run() [node] 9: 0xb72074 [node] 10: 0xb75657 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [node] 11: 0x154c4b6 [node] 12: 0x155ebe4 [node] 13: 0x154ce08 uv_run [node] 14: 0xa43975 node::SpinEventLoop(node::Environment*) [node] 15: 0xb4b0e6 node::NodeMainInstance::Run() [node] 16: 0xaccb3c node::Start(int, char**) [node] 17: 0x7fceee9a3083 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6] 18: 0xa408ec [node] image