search

CSB5 / OPERA-MS

OPERA-MS - Hybrid Metagenomic Assembler
Other
89 stars 17 forks source link

Error in during reference based clustering #25

Open Shirley-Q opened 5 years ago

Shirley-Q commented 5 years ago

Hi, I have an error in during reference based clustering when I run the OPERA-MS. I am not able to solve it. Thank you! Looking forward to your reply. Thank you very much! The content of 'sequence_similarity_clustering.err' is followed. /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/hierarchical_clustering/clusters

rm -r /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/ rm: cannot remove '/home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/': No such file or directory

mkdir -p /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH/

mkdir /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//INTER_FILES/

mkdir /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH/ Use of uninitialized value $contig_cluster in hash element at /home/emma/OPERA-MS/bin/sequence_similarity_clustering.pl line 888.

/home/emma/OPERA-MS//utils//mash paste /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//cluster.msh /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch ERROR: The file "/home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch" does not look like a sketch.

/home/emma/OPERA-MS//utils//mash dist -p 42 -d 0.90 /home/emma/OPERA-MS//genomeDB_Sketch.msh /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//cluster.msh > /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.dat 2> /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.err Error in during bin/mash dist. Please see /home/emma/QXJ/nanopore/trim_nanopore_out//intermediate_files/reference_clustering/MASH//mash_dist.err for details. The detail of 'mash_dist.err' is as followed. terminate called after throwing an instance of 'kj::ExceptionImpl' what(): src/kj/io.c++:42: failed: expected n >= minBytes; Premature EOF stack: 0x45ed59 0x4623aa 0x463ca7 0x46316d 0x45a736 0x42892b 0x42e603 0x417ad1 0x438ea2 0x425076 0x40377c 0x7fd5ec5a8830 0x403db1 Aborted (core dumped)

dbertran78 commented 5 years ago

Dear Shirley-Q, Can you please run the following command to create a complete log files: sh /home/emma/OPERA-MS/bin/get_log.sh /home/emma/QXJ/nanopore/trim_nanopore_out/ OPERA-MS-LOG.tar.gz will be create in your working directory.

Please send OPERA-MS-LOG.tar.gz, and the command line used for the OPERA-MS run to tong_chengxuan@gis.a-star.edu.sg and bertrandd@gis.a-star.edu.sg

Best regards,

--- Denis

ADSakura commented 4 years ago

Hi, I have a similar error of 'kj::ExceptionImpl' and the 'mash_dist.err' is as followed.

terminate called after throwing an instance of 'kj::ExceptionImpl' what(): src/capnp/serialize.c++:159: failed: expected segmentCount < 512; Message has too many segments. stack: 0x45ed59 0x4623aa 0x45ab26 0x42892b 0x42e603 0x417ad1 0x438ea2 0x425076 0x40377c 0x7f8a469a0830 0x403db1 Aborted (core dumped)

How did you solve this problem? Could you give me any advice about fixing this? Thanks.

dbertran78 commented 4 years ago

Dear ADSakura, We will need additional information to help you solve this issue. Could you please generate the complete log file using the OPERA-MS/bin/get_log.sh script. Please send the generated OPERA-MS-LOG.tar.gz, and the command line you used for the OPERA-MS run to tong_chengxuan@gis.a-star.edu.sg and bertrandd@gis.a-star.edu.sg

Best regards,

--- Denis

ValeriiaLadyhina commented 6 months ago

Hi!

I also got the error of reference based clustering and need help.

Here are the logs:

1) "mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A

version v0.9.0

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//megahit_v1.0.4-beta_LINUX_CPUONLY_x86_64-bin//megahit --num-cpu-threads 20 -1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R1.fastq.gz -2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R2.fastq.gz -o /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit.err

rm -r /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/intermediate_contigs

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS//OPERA-LG/bin//OPERA-long-read.pl --contig-file /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/final.contigs.fa --kmer 60 --long-read-file /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/minimap2/clean_from_host_genome/ready/Sample_1_5A_NP_host_removed.fastq --output-prefix opera --output-directory /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping --num-of-processors 20 --opera /export/opt/sw/opera-ms/0.8.3/OPERA-MS//OPERA-LG/bin/ --illumina-read1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R1.fastq.gz --illumina-read2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/bowtie2/clean_from_host_genome/Sample_1_5A_L001_host_removed_R2.fastq.gz --samtools-dir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --perl-dir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --minimap2 /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --short-read-tooldir /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ --skip-opera 1 --skip-short-read-mapping 1 > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/OPERA-long-read.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/OPERA-long-read.err

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation

cp /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/opera-ms-utils.config /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.config

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS//bin/cov_estimate.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.config > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/cov_estimate.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/filter_edge_coverage_ratio.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/contigs_340_80 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/filter_edge_coverage_ratio.err

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i0

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i0 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i0/pairedEdges_i0

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i1

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i1 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i1/pairedEdges_i1

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i2

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i2 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i2/pairedEdges_i2

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i3

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i3 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i3/pairedEdges_i3

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i4

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i4 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i4/pairedEdges_i4

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i5

ln -s /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping/pairedEdges_i5 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/pairedEdges_i5/pairedEdges_i5

/export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/sigma /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.config > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/sigma.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/refine_r_estimate.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/coverage_estimation/contigs_340_80 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ 20 6000000 /export/opt/sw/opera-ms/0.8.3/OPERA-MS/ 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/refine_r.err

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/perl /export/opt/sw/opera-ms/0.8.3/OPERA-MS/bin/sequence_similarity_clustering.pl /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/read_mapping /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/megahit_assembly/final.contigs.fa 20 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB /export/opt/sw/opera-ms/0.8.3/OPERA-MS/ /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms/ /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//MUMmer3.23/ NULL 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/sequence_similarity_clustering.err

Error in during reference based clustering. Please see /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/sequence_similarity_clustering.err for details."

2) "cat sequence_similarity_clustering.err /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/hierarchical_clustering/clusters

rm -r /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/ rm: cannot remove ‘/proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/’: No such file or directory

mkdir -p /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH/

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//INTER_FILES/

mkdir /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH/

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash sketch -p 20 -k 21 -s 1000 -o /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//INTER_FILES//* > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_sketch.out 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_sketch.err

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err sh: line 1: 6795 Floating point exception(core dumped) /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err Error in during bin/mash dist. Please see /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err for details."

3) But file mash_dist.err doesn't have any information.

Can you please help me out?

Best regards, Valeriia

jsgounot commented 6 months ago

Hi Valeriia,

please open a new issue, your error message is not the same. Here you have a weird floating point exception error with mash:

sh: line 1: 6795 Floating point exception(core dumped) /export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat 2> /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist.err

I can't know exactly what's happening here, but I would guess a memory error. How much memory were you giving here?

Can you try to increase this value and rerun this command line alone?

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -p 20 -d 0.90 /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS_DB/genomes.msh /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//PARTIAL_SKETCH//partial_Sketch0.msh > /proj/pig_amresistance/NOBACKUP/Pilot_study/Tem_comp/OPERA_MS/Sample_1_5A/intermediate_files/reference_clustering/MASH//mash_dist_0.dat

Can you also try to run mash help command to see if this is not a compilation/system issue?

/export/opt/sw/opera-ms/0.8.3/OPERA-MS//tools_opera_ms//mash dist -h

Regards, JSG