SNAP failed with Transrate v 1.0.3

[Jinjie-Duan]

Hi,

I got the following error when running transrate v 1.0.3

[ERROR] 2017-02-23 16:07:32 : Snap failed Welcome to SNAP version 1.0dev.96.

mmap failed. MemMapDataReader: fail to map /home/jjduan/python/01highQualityData/withoutN-trimlastbase/110902_SN132_B_L002_R2_GLQ-13b.fastq at 0,1966390515 SNAP exited with exit code 1 from line 2283 of file SNAPLib/DataReader.cpp

I saw a previous issues #190 asked the same question but with older version of transrate and @blahah solved this bug in version 1.0.3. But I still get the same error when running verion 1.0.3.

Any suggestions? Many thanks in advance.

best wishes, Jinjie

[lgleason]

I am having this same issue. Jinjie, did you ever figure out a way to get around this error?

Thanks! Lani

[blahah]

@Jinjie-Duan thanks for opening a new issue here, and @lgleason thanks for reminding me!

If either of you are able to upload somewhere a set of reads (as small as possible) that causes this error, that would really help us get it fixed.

[blahah]

@Jinjie-Duan @lgleason actually, after reading the SNAP code for the function that is errorring, it seems most likely to me that you're running out of memory.

Please could you try running the command again, and this time watching the memory of the process? On most linux systems you can do this by putting /usr/bin/time -v before the transrate command.

Please could you post the output of the above, plus info about your available RAM (on linux you can try cat /proc/meminfo | grep Total)?

[lgleason]

Hi Richard,

Thanks for the response. I'm currently running the command again (I allocated 240gb of memory this time), and concurrently watching the memory of the process, and will post an update with the output. I'm actually running this on a cluster...so far I have tried with various amounts of memory; most recently, I got the same error with 120gb of RAM. Do you have a vague idea of how much memory is necessary? I have about 50 PE50 Illumina reads.

Thanks, Lani

On Wed, May 10, 2017 at 9:42 AM, Richard Smith-Unna < notifications@github.com> wrote:

@Jinjie-Duan https://github.com/Jinjie-Duan @lgleason https://github.com/lgleason actually, after reading the SNAP code for the function that is errorring https://github.com/amplab/snap/blob/master/SNAPLib/DataReader.cpp#L2282-L2318, it seems most likely to me that you're running out of memory.

Please could you try running the command again, and this time watching the memory of the process? On most linux systems you can do this by putting /usr/bin/time -v before the transrate command.

Please could you post the output of the above, plus info about your available RAM (on linux you can try cat /proc/meminfo | grep Total)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/blahah/transrate/issues/206#issuecomment-300542024, or mute the thread https://github.com/notifications/unsubscribe-auth/AHZCCDY6RbTR8my52fTBGHjvJT9_1iBtks5r4ejugaJpZM4MLLSz .

[lgleason]

Even with 240gb of memory, I am still getting the same error, output below:

"ERROR] 2017-05-24 14:38:24 : Snap failed Welcome to SNAP version 1.0dev.96.

mmap failed. MemMapDataReader: fail to map /N/dc2/projects/DowdMCali/PE50/49w_1.fq at 0,6325218679 SNAP exited with exit code 1 from line 2283 of file SNAPLib/DataReader.cpp

Command exited with non-zero status 1 Command being timed: "/N/u/lgleaso/Mason/local/transrate/transrate --assembly Trinity_out/PSC_assemblies/Trinity_merged_noSpacenoComma_C32.fasta --output Trinity_out/PSC_assemblies/Transrate_allAssembly --left PE150/2e_R1.fq,PE150/38p_R1.fq,PE150/46b_R1.fq,$ User time (seconds): 2925.89 System time (seconds): 43.99 Percent of CPU this job got: 99% Elapsed (wall clock) time (h:mm:ss or m:ss): 49:55.90 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 10376648 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 2524646 Voluntary context switches: 6567 Involuntary context switches: 301543 Swaps: 0 File system inputs: 8 File system outputs: 8 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1"

Any ideas to fix this? Just continue to increase the memory allocated to the job?

Thank you! Lani

On Wed, May 24, 2017 at 8:45 AM, Lani Gleason lani.gleason@gmail.com wrote:

Hi Richard,

Thanks for the response. I'm currently running the command again (I allocated 240gb of memory this time), and concurrently watching the memory of the process, and will post an update with the output. I'm actually running this on a cluster...so far I have tried with various amounts of memory; most recently, I got the same error with 120gb of RAM. Do you have a vague idea of how much memory is necessary? I have about 50 PE50 Illumina reads.

Thanks, Lani

On Wed, May 10, 2017 at 9:42 AM, Richard Smith-Unna < notifications@github.com> wrote:

@Jinjie-Duan https://github.com/Jinjie-Duan @lgleason https://github.com/lgleason actually, after reading the SNAP code for the function that is errorring https://github.com/amplab/snap/blob/master/SNAPLib/DataReader.cpp#L2282-L2318, it seems most likely to me that you're running out of memory.

Please could you try running the command again, and this time watching the memory of the process? On most linux systems you can do this by putting /usr/bin/time -v before the transrate command.

Please could you post the output of the above, plus info about your available RAM (on linux you can try cat /proc/meminfo | grep Total)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/blahah/transrate/issues/206#issuecomment-300542024, or mute the thread https://github.com/notifications/unsubscribe-auth/AHZCCDY6RbTR8my52fTBGHjvJT9_1iBtks5r4ejugaJpZM4MLLSz .

[blahah]

According to the output the process only used ~10gb of RAM, so it looks like the problem is a bug in SNAP or an issue with the input data. To confirm whether it's something specific about this file, could run transrate with just the 49* pair of read files?

[lgleason]

When I run transrate with just the 49* pair of files, I get no errors and everything runs normally.

When I use all original input files and increase the allocated memory to 480gb, I get a new error:

[ERROR] 2017-06-08 11:53:30 : Snap failed Welcome to SNAP version 1.0dev.96.

FASTQ file doesn't end with a newline! Failing. fileOffset = 801439481, validBytes = 263 SNAP exited with exit code 1 from line 252 of file SNAPLib/FASTQ.cpp

Command exited with non-zero status 1 Command being timed: "/N/u/lgleaso/Mason/local/transrate/transrate --assembly Trinity_out/PSC_assemblies/Trinity_merged_noSpacenoComma_C32.fasta --output Trinity_out/PSC_assemblies/Transrate_allAssembly --left PE150/2e_R1.fq,PE150/38p_R1.fq,PE150/46b_R1.fq,$ User time (seconds): 50656.58 System time (seconds): 153.70 Percent of CPU this job got: 583% Elapsed (wall clock) time (h:mm:ss or m:ss): 2:25:03 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 48369996 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 10254 Minor (reclaiming a frame) page faults: 4060094 Voluntary context switches: 140547 Involuntary context switches: 5449570 Swaps: 0 File system inputs: 280 File system outputs: 1530560 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1

Is there a way to tell which specific FASTQ file it's referring to?

Thank you! Lani

On Thu, May 25, 2017 at 3:09 AM, Richard Smith-Unna < notifications@github.com> wrote:

According to the output the process only used ~10gb of RAM, so it looks like the problem is a bug in SNAP or an issue with the input data. To confirm whether it's something specific about this file, could run transrate with just the 49* pair of read files?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/blahah/transrate/issues/206#issuecomment-303974351, or mute the thread https://github.com/notifications/unsubscribe-auth/AHZCCBr-VJESUSUv8Uo9t5npihkImgg0ks5r9VNDgaJpZM4MLLSz .

[blahah]

Something like this (run in a directory that only contains the fq files) maybe:

find -type f -print0 | xargs -0 -L1 bash -c 'test "$(tail -c 1 "$0")" && echo "No new line at end of $0"'

Then for any files listed do:

echo '' >> filename