Test data fails at gap-filling step, returns to prompt, leaves core file and empty consensus_cmd.sh.log file

[RachelRodgers]

I have installed the latest version of OPERA-MS (installed today, Feb 22, 2021) on an Ubuntu 16.04.6 cluster. Using an interactive session to run the test files with the following commands (I did not submit a job to the SLURM scheduler yet):

cd OPERA-MS/test_files
ml perl/5.26.2
ml jdk/11.0.2_9
perl ../OPERA-MS.pl \
--contig-file contigs.fasta \
--short-read1 R1.fastq.gz \
--short-read2 R2.fastq.gz \
--long-read long_read.fastq \
--no-ref-clustering \
--out-dir RESULTS 2> log.err

The program runs until the Gap filling step and fails, returning to prompt. Following error logs leads to an empty error log file. A core file is also written out. Here is the output I see (running after a failed attempt):

[Mon Feb 22 15:59:54 2021]      Short read assembly [1/9]
[Mon Feb 22 15:59:54 2021]      Skip [contig file provided as input]

[Mon Feb 22 15:59:55 2021]      Long-read mapping and assembly graph generation [2/9]
[Mon Feb 22 15:59:55 2021]      Stage previously completed

[Mon Feb 22 15:59:55 2021]      Short-read mapping and coverage estimation [3/9]
[Mon Feb 22 15:59:55 2021]      Stage previously completed

[Mon Feb 22 15:59:55 2021]      Hierarchical clustering [4/9]
[Mon Feb 22 15:59:55 2021]      Stage previously completed

[Mon Feb 22 15:59:55 2021]      Reference based clustering [5/9]
[Mon Feb 22 15:59:55 2021]      Skip

[Mon Feb 22 15:59:55 2021]      Strain clustering and assembly [6/9]
[Mon Feb 22 15:59:55 2021]      Skip

[Mon Feb 22 15:59:55 2021]      Assembly of other clusters [7/9]
[Mon Feb 22 15:59:55 2021]      Stage previously completed

[Mon Feb 22 15:59:55 2021]      Gap filling [8/9]

log.err reads:

mkdir -p /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS/

version v0.9.0

rm -r /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/user_assembly

mkdir -p /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/user_assembly

rm -r /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING

mkdir -p /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING

/scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms/perl /scratch/mtblab/rachel/OPERA-MS/test_files/..//OPERA-LG/bin//gapfilling.pl /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/user_assembly/contigs.fasta /scratch/mtblab/rachel/OPERA-MS/test_files/long_read.fastq /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/read_mapping 2 499 /scratch/mtblab/rachel/OPERA-MS/test_files/..//OPERA-LG/bin/ /scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms/ /scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms/ /scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms/ /scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms//MUMmer3.23/ 2> /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/gap_filling.err

Error in during gap_filling. Please see /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/gap_filling.err for details.

gap_filling.err reads:

/scratch/mtblab/rachel/OPERA-MS/test_files/..//tools_opera_ms//perl /scratch/mtblab/rachel/OPERA-MS/test_files/..//OPERA-LG/bin//extract_read_sequence.pl --edge-file /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/read_mapping/edge_read_info.dat --contig-file /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/user_assembly/contigs.fasta --opera-lr-dir /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/../ --read-file /scratch/mtblab/rachel/OPERA-MS/test_files/long_read.fastq --output-directory /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING 2> /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/extract_read.err

cat /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/consensus_cmd.sh | xargs -L 1 -P 2 -I COMMAND sh -c "COMMAND" 2> /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/consensus_cmd.sh.log

Error during consensus sequence generation. Please see /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/consensus_cmd.sh.log for details.

and consensus_cmd.sh.log is empty.

Please let me know if there is additional information I can provide, such as the core file, etc.

Thanks!

[jsgounot]

Hello,

thanks for trying OPERA-MS and sorry for the late reply. What happen if you run the specific command line directly from the terminal : cat /scratch/mtblab/rachel/OPERA-MS/test_files/RESULTS//intermediate_files/opera_long_read/GAPFILLING/consensus_cmd.sh | xargs -L 1 -P 2 -I COMMAND sh -c "COMMAND" ?

[RachelRodgers]

Hi, thanks for the reply. Using 1GB RAM and 1 CPU this command runs for several seconds before return to prompt, and dumps a 584K core file.

[jsgounot]

Hi. You need to use at least 2 CPU (-P 2). If it's still not working, could you also check the same command with more memory (8G should be fine) ? Thanks !

[RachelRodgers]

Hi, sorry for the late reply. I'm still having the same issue. Can I send you the core file?

[jsgounot]

Yes sure.

[RachelRodgers]

Thanks. Are you able to access it at this link?

https://wustl.box.com/s/72odr8ubc2lkpp9zcmxy28c64sw2nmn4

[jsgounot]

Hi, sadly there is not much I've been able to do with this file. Apparently, this occurs most of the time when your job does not have enough computer resources. Did you try to increase the memory size too ?

[RachelRodgers]

Yes, I tried increasing memory too, but I will continue to increase the resources and see what happens. In the past I have had issues with software compiled for certain CPU architectures behaving in the same way (crashing with core file). Any chance that is an issue here? I would doubt it, as I have colleagues running other versions of the software on the same cluster, but thought I would ask. Thanks for your help!