Getting 'Critical exception: The length of the requested memory exceeds the CHUNK_SIZE' while running Ray

[PolinaN]

I keep getting this error while running Ray

Rank 30: assembler memory usage: 342492 KiB Critical exception: The length of the requested memory exceeds the CHUNK_SIZE: 4718592 > 4194304 Ray: RayPlatform/memory/MyAllocator.cpp:97: void* MyAllocator::allocate(int): Assertion `false' failed. [r1i0n6:20031] * Process received signal * [r1i0n6:20031] Signal: Aborted (6) [r1i0n6:20031] Signal code: (-6) [r1i0n6:20031] [ 0] /lib64/libpthread.so.0(+0xf6b0) [0x2aaaabb0a6b0] [r1i0n6:20031] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x2aaaacd32945] [r1i0n6:20031] [ 2] /lib64/libc.so.6(abort+0x181) [0x2aaaacd33f21] [r1i0n6:20031] [ 3] /lib64/libc.so.6(__assert_fail+0xf0) [0x2aaaacd2b810] [r1i0n6:20031] [ 4] Ray() [0x531fa8] [r1i0n6:20031] [ 5] Ray() [0x4e94bf] [r1i0n6:20031] [ 6] Ray() [0x4e63b1] [r1i0n6:20031] [ 7] Ray() [0x552dff] [r1i0n6:20031] [ 8] Ray() [0x554277] [r1i0n6:20031] [ 9] Ray() [0x4127f4] [r1i0n6:20031] [10] Ray() [0x41068b] [r1i0n6:20031] [11] /lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaacd1ebc6] [r1i0n6:20031] [12] Ray() [0x40d811]

[r1i0n6:20031] * End of error message *

mpirun noticed that process rank 31 with PID 20031 on node r1i0n6.ice.gmi.oeaw.ac.at exited on signal 6 (Aborted).

how can I specify maximum CHUNK_SIZE?

Thanks, Polina

[sebhtml]

On 16/11/13 10:03 AM, PolinaN wrote:

I keep getting this error while running Ray

Rank 30: assembler memory usage: 342492 KiB Critical exception: The length of the requested memory exceeds the CHUNK_SIZE: 4718592 > 4194304

Are you using super long reads ?

Ray: RayPlatform/memory/MyAllocator.cpp:97: void* MyAllocator::allocate(int): Assertion `false' failed. [r1i0n6:20031] * Process received signal * [r1i0n6:20031] Signal: Aborted (6) [r1i0n6:20031] Signal code: (-6) [r1i0n6:20031] [ 0] /lib64/libpthread.so.0(+0xf6b0) [0x2aaaabb0a6b0] [r1i0n6:20031] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x2aaaacd32945] [r1i0n6:20031] [ 2] /lib64/libc.so.6(abort+0x181) [0x2aaaacd33f21] [r1i0n6:20031] [ 3] /lib64/libc.so.6(_assert_fail+0xf0) [0x2aaaacd2b810] [r1i0n6:20031] [ 4] Ray() [0x531fa8] [r1i0n6:20031] [ 5] Ray() [0x4e94bf] [r1i0n6:20031] [ 6] Ray() [0x4e63b1] [r1i0n6:20031] [ 7] Ray() [0x552dff] [r1i0n6:20031] [ 8] Ray() [0x554277] [r1i0n6:20031] [ 9] Ray() [0x4127f4] [r1i0n6:20031] [10] Ray() [0x41068b] [r1i0n6:20031] [11] /lib64/libc.so.6(_libc_start_main+0xe6) [0x2aaaacd1ebc6] [r1i0n6:20031] [12] Ray() [0x40d811]

[r1i0n6:20031] *** End of error message ***

mpirun noticed that process rank 31 with PID 20031 on node r1i0n6.ice.gmi.oeaw.ac.at exited on signal 6 (Aborted).

how can I specify maximum CHUNK_SIZE?

Thanks, Polina

— Reply to this email directly or view it on GitHub https://github.com/sebhtml/ray/issues/219.

[PolinaN]

yes, mean length - 18105bp, maximum length - 63987 bp (which is supposed to be less than maximum http://comments.gmane.org/gmane.science.biology.ray-genome-assembler/594)

[PolinaN]

btw, I use MAXKMERLENGTH=250 in this case. Can it cause the problem?

[pforai]

FYI We've also compiled the software with FORCE_PACKING=1 if that makes any difference.

[sebhtml]

FORCE_PACKING removes padding in the structures, but may also cause problems on some architectures (Itanium or Sparc for instance). On x86-64, I don't think FORCE_PACKING=1 is a problem.

[jjkoehorst]

I have the same issue with reads of 326 nucleotides long. Another dataset with 300bp long runs just fine. Any solution to this particular problem?

[sebhtml]

Hi,

I am not sure how can this happen since this memory pool is used to allocate memory for reads.

Are you providing genomes in input ?

[bastian-wur]

Bumping this, although, I know, is quite some time ago. Was same lab as @jjkoehorst btw, now different one, but experienced this one also before once, I think.

Error:

Rank 8: gossiping generated 0 messages (gossips: 0 ---> 0)
Rank 9: gossiping generated 0 messages (gossips: 0 ---> 0)
Critical exception: The length of the requested memory exceeds the CHUNK_SIZE: 49221256 > 33554432
Ray: RayPlatform/memory/MyAllocator.cpp:97: void* MyAllocator::allocate(int): Assertion `false' failed.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 134
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions

Now, the weird thing: I run Ray in an assembly pipeline. I did multiple runs with the pipeline, and the pipeline will try to assemble again, if no result is present. Therefore, I can see at the moment that a) this is reproducible, at least on the same dataset b) it doesn't apply to all k-mers. I run it with kmers between 51 and 151 (yes, compiled with higher limit). This leads to a) genome a assembles with kmers 101, 111, 121, 131, 141 b) genome b assembles with kmers 51, 111, 121, 131, 141 c) genome c assembles with kmer 141

Actually, a and c are the same genome. But 2 different runs, where c has 4 times the amount of reads (think the sequencing supplier messed up something). Also genome b is nearly the same, there are only a few generation times with some evolutionary pressure on it in between. All the data is originally HiSeq4000 150bp PE (but after trimming).

This happens with 2.3.0 and 2.3.1, on Ubuntu 14.04.5 LTS on a cluster. The actual RAM requirements cannot be the problem (since it refers to memory), since I schedule in the SGE like 90GB, but according to the report at the end the run doesn't exceed 30GB.

At the end, this mainly bothers me because Ray with kmer of 101 seems to give the best results (we have extremely close reference genomes available, so I can say this with some confidence), but I cannot get it to run with 101 on all the datasets :/.