search

artic-network / rampart

Read Assignment, Mapping, and Phylogenetic Analysis in Real Time
GNU General Public License v3.0
79 stars 34 forks source link

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory #77

Closed Psy-Fer closed 4 years ago

Psy-Fer commented 4 years ago

G'day,

Love your work.

I'm getting a core dump on a run. It's 2 samples, on a MinION flowcell, running on a GridION.

I got the error once while it was running normally, and then again once I restarted it. Here are the tails of the verbose output.

[verbose]  [annotation parser]      Parsing annotation for FAM92219_pass_8eb702dd_1111
[verbose]  [annotation parser]      0 files remain in queue
[verbose]  [pipeline (Annotate reads)]queue empty, processed 1243 files
[verbose]  [annotation parser]      parsed annotation file, FAM92219_pass_8eb702dd_1111 (4000 lines)
[verbose]  [socket]                 sendData
[verbose]  [Timer getDataForClient]  #1260 time: 21ms. average: 9ms.

<--- Last few GCs --->

[13338:0x559f99cbe870] 20958751 ms: Mark-sweep 2044.2 (2067.0) -> 2044.2 (2067.8) MB, 1380.0 / 0.0 ms  (average mu = 0.098, current mu = 0.035) allocation failure scavenge might not succeed
[13338:0x559f99cbe870] 20960237 ms: Mark-sweep 2045.2 (2067.8) -> 2044.8 (2087.8) MB, 1479.4 / 0.0 ms  (average mu = 0.051, current mu = 0.004) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7fb2bddd85c9 node::Abort() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 2: 0x7fb2bddd9296 node::OnFatalError(char const*, char const*) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 3: 0x7fb2be00a0c2 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 4: 0x7fb2be00a37b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 5: 0x7fb2be1942a6  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 6: 0x7fb2be1a7221 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 7: 0x7fb2be1a7f2d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 8: 0x7fb2be1a831d v8::internal::Heap::CollectAllGarbage(int, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 9: 0x7fb2be1ac08e v8::internal::IncrementalMarkingJob::Task::Step(v8::internal::Heap*, v8::EmbedderHeapTracer::EmbedderStackState) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
10: 0x7fb2be1ac190 v8::internal::IncrementalMarkingJob::Task::RunInternal() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
11: 0x7fb2be0f3156  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
12: 0x7fb2bde47cb2 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
13: 0x7fb2bde4a445 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
14: 0x7fb2bdf9e771  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
15: 0x7fb2bdfaea76  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
16: 0x7fb2bdf9f072 uv_run [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
17: 0x7fb2bde1cab7 node::NodeMainInstance::Run() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
18: 0x7fb2bdda7ef8 node::Start(int, char**) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
19: 0x559f98a6f117 main [node]
20: 0x7fb2bd173830 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
21: 0x559f98a6f172  [node]
Aborted (core dumped)

and the second after I ran the command again and it parsed all the existing files

Warning: Very few reads have mapped (less than 5%).

[verbose]  [annotation parser]      Parsing annotation for FAM92219_fail_8eb702dd_137
[verbose]  [annotation parser]      0 files remain in queue
[verbose]  [pipeline (Annotate reads)]queue length: 357, processed 7 files
[verbose]  [pipeline (Annotate reads)]snakemake --snakefile /home/grid/rampart/default_protocol/pipelines/demux_map/Snakefile --configfile /home/grid/rampart/default_protocol/pipelines/demux_map/config.yaml --config references_file=/home/grid/SARS-CoV-2_GTG/protocols/artic/V2/references.fasta limit_barcodes_to=NB11,NB12 input_path=/data/GLFL037928/GLFL037928/20200407_0830_X3_FAM92219_c0bff9d3/fastq_fail output_path=/data/SARS-CoV-2/artic-1/annotations/ filename_stem=FAM92219_fail_8eb702dd_138 --nolock --rerun-incomplete
[verbose]  [annotation parser]      parsed annotation file, FAM92219_fail_8eb702dd_137 (4000 lines)
[verbose]  [socket]                 sendData
[verbose]  [Timer getDataForClient]  #1250 time: 38ms. average: 11ms.
[verbose]  [pipeline (Annotate reads)]Proportion unmapped is 5.858546168958743

Warning: Very few reads have mapped (less than 5%).

[verbose]  [annotation parser]      Parsing annotation for FAM92219_fail_8eb702dd_138
[verbose]  [annotation parser]      0 files remain in queue
[verbose]  [pipeline (Annotate reads)]queue length: 356, processed 8 files
[verbose]  [pipeline (Annotate reads)]snakemake --snakefile /home/grid/rampart/default_protocol/pipelines/demux_map/Snakefile --configfile /home/grid/rampart/default_protocol/pipelines/demux_map/config.yaml --config references_file=/home/grid/SARS-CoV-2_GTG/protocols/artic/V2/references.fasta limit_barcodes_to=NB11,NB12 input_path=/data/GLFL037928/GLFL037928/20200407_0830_X3_FAM92219_c0bff9d3/fastq_fail output_path=/data/SARS-CoV-2/artic-1/annotations/ filename_stem=FAM92219_fail_8eb702dd_139 --nolock --rerun-incomplete
[verbose]  [annotation parser]      parsed annotation file, FAM92219_fail_8eb702dd_138 (4000 lines)
[verbose]  [socket]                 sendData
[verbose]  [Timer getDataForClient]  #1251 time: 47ms. average: 11ms.

<--- Last few GCs --->

[29997:0x55707ba5c870]  1190245 ms: Mark-sweep 2031.4 (2053.8) -> 2031.3 (2054.6) MB, 1334.7 / 0.0 ms  (average mu = 0.079, current mu = 0.005) allocation failure scavenge might not succeed
[29997:0x55707ba5c870]  1191533 ms: Mark-sweep 2032.1 (2054.6) -> 2032.1 (2055.6) MB, 1283.2 / 0.0 ms  (average mu = 0.043, current mu = 0.004) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3ce55eddb3a9 <JSObject>
    0: builtin exit frame: stringify(this=0x3ce55edcaf01 <Object map = 0x37082cd81431>,0x1f48b4d004b9 <undefined>,0x1f48b4d004b9 <undefined>,0x39e90fa38449 <JSArray[2]>,0x3ce55edcaf01 <Object map = 0x37082cd81431>)

    1: arguments adaptor frame: 1->3
    2: tryStringify [0x29aacf547ed1] [/home/grid/rampart/node_modules/socket.io-parser/index.js:184] [bytecode=0x1c80c1afc019 offset=15](this=0x2...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7fd8d2a675c9 node::Abort() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 2: 0x7fd8d2a68296 node::OnFatalError(char const*, char const*) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 3: 0x7fd8d2c990c2 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 4: 0x7fd8d2c9937b v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 5: 0x7fd8d2e232a6  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 6: 0x7fd8d2e36221 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 7: 0x7fd8d2e36f2d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 8: 0x7fd8d2e39294 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
 9: 0x7fd8d2e396d6 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
10: 0x7fd8d2e01e5b v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
11: 0x7fd8d2e08f4f v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
12: 0x7fd8d318a468 v8::internal::IncrementalStringBuilder::Extend() [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
13: 0x7fd8d2f0beb8 v8::internal::JsonStringifier::SerializeDouble(double) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
14: 0x7fd8d2f10394 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
15: 0x7fd8d2f0fd21 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
16: 0x7fd8d2f0fd21 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
17: 0x7fd8d2f0fd21 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
18: 0x7fd8d2f126ad v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
19: 0x7fd8d2f13722 v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
20: 0x7fd8d2f123d7 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
21: 0x7fd8d2f13e2f v8::internal::JsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
22: 0x7fd8d2f13ebc v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
23: 0x7fd8d2d3cf02  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
24: 0x7fd8d2d3d739 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
25: 0x7fd8d36390b9  [/home/grid/miniconda3/envs/artic-rampart/bin/../lib/libnode.so.79]
Aborted (core dumped)

Any thoughts one what's causing this, and how to fix?

Let me know if you need any more info.

Cheers, James

jameshadfield commented 4 years ago

Hey James -- can you confirm this is rampart version 1.1.0 you're running? If so, is there any way you could share the fastqs with me, as I don't have any datasets which do this (DM hamesjadfield on twitter if needed). Cheers.

Psy-Fer commented 4 years ago

Hey James, it's 1.1.0-prerelease

Think it might need an update?

jameshadfield commented 4 years ago

Yes, if you update to 1.1.0 (see https://community.artic.network/t/rampart-v1-1-0-released/69/6) this should be fixed. Please do let me know if it's still occuring as it's clearly a rather sizable problem!

Psy-Fer commented 4 years ago

I'll give it a shot later tonight when this other run is done and let you know when I test it out.

Cheers.

jameshadfield commented 4 years ago

@Psy-Fer this should have been fixed in v1.1.0 (i've just pushed v1.2.0). Please reopen if you see this again!

Psy-Fer commented 4 years ago

Hey dude,

Sorry, I updated to 1.1.0 2 days ago with some new gridION samples, using our 2.5kb amplicons and the nanopore rapid kit, and it worked perfectly.

Awesome work dude! Loving it.