search

bxlab / metaWRAP

MetaWRAP - a flexible pipeline for genome-resolved metagenomic data analysis
MIT License
393 stars 191 forks source link

Controlled exit resulting from an unrecoverable error or warning #493

Open Valentin-Bio opened 1 year ago

Valentin-Bio commented 1 year ago

Hello I previously ran metawrap bin refinement module using a different dataset and everything went good. This time I'm running it over a different dataset but the process fails.

dir="/path-to-/workingdir" metawrap bin_refinement -o ${dir}/refined_mags -t 32 -A ${dir}/metabat2 -B ${dir}/concoct/bins -C ${dir}/vamb/bins -x 10 -c 50

Reading the log file I noticed that the program removes a directory that has not refined bins 'Removing bin set binsA.tmp because it yielded 0 refined bins ...'

after removing the content under the above mentioned directory check starts over the removed dataset:

[2023-03-15 11:08:35] INFO: CheckM v1.0.18 [2023-03-15 11:08:35] INFO: checkm lineage_wf -x fa binsA binsA.checkm -t 32 --tmpdir binsA.tmp --pplacer_threads 1 [2023-03-15 11:08:35] INFO: [CheckM - tree] Placing bins in reference genome tree.

Controlled exit resulting from an unrecoverable error or warning.

How can I deal with this problem ? seems like there is nothing wrong on the provided input fasta files but the program removes intermediate directories that has not refined bins.

Here is the full log:

metawrap bin_refinement -o /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/refined_mags -t 32 -A /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/metabat2 -B /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins -C /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/vamb/bins -x 10 -c 50

------------------------------------------------------------------------------------------------------------------------
-----            There is 40 RAM and 32 threads available, and each pplacer thread uses >40GB, so I will           -----
-----                                          use 1 threads for pplacer                                           -----
------------------------------------------------------------------------------------------------------------------------

########################################################################################################################
#####                                                BEGIN PIPELINE!                                               #####
########################################################################################################################

------------------------------------------------------------------------------------------------------------------------
-----                               setting up output folder and copying over bins...                              -----
------------------------------------------------------------------------------------------------------------------------

************************************************************************************************************************
*****             Warning: /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/refined_mags already             *****
*****                                         exists. Attempting to clean.                                         *****
************************************************************************************************************************

------------------------------------------------------------------------------------------------------------------------
-----                                          there are 150 bins in binsA                                         -----
------------------------------------------------------------------------------------------------------------------------

Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/105.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/110.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/120.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/132.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/133.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/134.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/151.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/156.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/160.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/19.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/24.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/27.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/30.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/47.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/54.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/56.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/71.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/72.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/73.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/74.fa because the bin size is not between 50kb and 20Mb
Skipping /lustre/groups/cbi/Users/ecastron/valentin/diesel_mags/concoct/bins/76.fa because the bin size is not between 50kb and 20Mb

------------------------------------------------------------------------------------------------------------------------
-----                                          there are 144 bins in binsB                                         -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                          there are 159 bins in binsC                                         -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                             There are 3 bin sets!                                            -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                              Fix contig naming by removing special characters...                             -----
------------------------------------------------------------------------------------------------------------------------

########################################################################################################################
#####                                             BEGIN BIN REFINEMENT                                             #####
########################################################################################################################

------------------------------------------------------------------------------------------------------------------------
-----              There are three bin folders, so there 4 ways we can refine the bins (A+B, B+C, A+C,             -----
-----                                    A+B+C). Will try all four in parallel!                                    -----
------------------------------------------------------------------------------------------------------------------------

Specified 2 input bin sets: -1 binsC -2 binsB
Add folder/bin name to contig name for binsC bins
Add folder/bin name to contig name for binsB bins
Combine all bins together
The number of refined bins: 79
Exporting refined bins...
Extracting refined bin: Refined_79.fasta
Deleting temporary files

All done!
Specified 2 input bin sets: -1 binsA -2 binsC
Add folder/bin name to contig name for binsA bins
Add folder/bin name to contig name for binsC bins
Combine all bins together
The number of refined bins: 81
Exporting refined bins...
Extracting refined bin: Refined_81.fasta
Deleting temporary files

All done!
Specified 3 input bin sets: -1 binsA -2 binsB -3 binsC
Add folder/bin name to contig name for binsA bins
Add folder/bin name to contig name for binsB bins
Add folder/bin name to contig name for binsC bins
Combine all bins together
The number of refined bins: 81
Exporting refined bins...
Extracting refined bin: Refined_81.fasta
Deleting temporary files

All done!
Specified 2 input bin sets: -1 binsA -2 binsB
Add folder/bin name to contig name for binsA bins
Add folder/bin name to contig name for binsB bins
Combine all bins together
The number of refined bins: 125
Exporting refined bins...
Extracting refined bin: Refined_125.fasta
Deleting temporary files

All done!

------------------------------------------------------------------------------------------------------------------------
-----                                     there are 125 refined bins in binsAB                                     -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                      there are 79 refined bins in binsBC                                     -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                      there are 81 refined bins in binsAC                                     -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                     there are 81 refined bins in binsABC                                     -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                                     Bin refinement finished successfully!                                    -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                            fixing bin naming to .fa convention for consistancy...                            -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                              making sure every refined bin set contains bins...                              -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                     Removing bin set binsA.checkm because it yielded 0 refined bins ...                      -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                       Removing bin set binsA.tmp because it yielded 0 refined bins ...                       -----
------------------------------------------------------------------------------------------------------------------------

########################################################################################################################
#####                                      RUNNING CHECKM ON ALL SETS OF BINS                                      #####
########################################################################################################################

------------------------------------------------------------------------------------------------------------------------
-----                                         Running CheckM on binsA bins                                         -----
------------------------------------------------------------------------------------------------------------------------

[2023-03-15 11:08:35] INFO: CheckM v1.0.18
[2023-03-15 11:08:35] INFO: checkm lineage_wf -x fa binsA binsA.checkm -t 32 --tmpdir binsA.tmp --pplacer_threads 1
[2023-03-15 11:08:35] INFO: [CheckM - tree] Placing bins in reference genome tree.

  Controlled exit resulting from an unrecoverable error or warning.

************************************************************************************************************************
*****                             Something went wrong with running CheckM. Exiting...                             *****
************************************************************************************************************************
nelsonruth11 commented 1 year ago

Did you ever find a fix for this? I'm running into the same issue