MetaWRAP assembly, binning and bin refinement not working

[sarahisherb]

Me again...trying to re-run the refinement step with less stringent sequencing and I get the following errors:

Here is what I tried to run

module load singularity mugqic/BBMap/38.90

bash ILL_pipelines/generateslurm_assembly_bin_refinement.metawrap.sh \
--sample_tsv 20220825_boreal_moss/samples.tsv \
--out mag_run2 \
--assembly --binning \
--refinement --refinement_min_compl 30 --refinement_max_cont 10

sbatch --array=1-187 mag_run2/assembly_bin_refinement.metawrap.slurm.sh

Copied and pasted from the github directly with changes, I think it looks all correct, but maybe I did something wrong still. Sorry!!

[jflucier]

1) you should run you test on a small sample set.

sbatch --array=1-10 mag_run2/assembly_bin_refinement.metawrap.slurm.sh

once you validate those 10 sample work you can launch the remaining samples:

sbatch --array=11-187 mag_run2/assembly_bin_refinement.metawrap.slurm.sh

2) you can access slurm logs for debugging. By default its goes in a subfolder where the results are stored. In your case these are stored ???/denovo_assembly/logs/ . I dont know where you are working but clearly not where I ran previous analysis (you should work there)

3) MOST IMPORTANT You say you want to rerun refinement step but in the command you tell soft to run assembly, binning and refinement step. Command should look like this:

bash ILL_pipelines/generateslurm_assembly_bin_refinement.metawrap.sh \
--sample_tsv 20220825_boreal_moss/samples.tsv \
--out mag_run2 \
--refinement --refinement_min_compl 30 --refinement_max_cont 10

The assembly and binning step take a long time to execute on fat node. You already have the results in /nfs3_ib/ip29-ib/ip29/ilafores_group/analysis/20220825_boreal_moss, no need to rerun.

To make this run, do the following:

newgrp def-ilafores
cd /nfs3_ib/ip29-ib/ip29/ilafores_group/analysis/20220825_boreal_moss
module load singularity mugqic/BBMap/38.90
bash /project/def-ilafores/common/ILL_pipelines/generateslurm_assembly_bin_refinement.metawrap.sh \
--sample_tsv 20220825_boreal_moss/samples.tsv \
--out mag_run2 \
--refinement --refinement_min_compl 30 --refinement_max_cont 10

[sarahisherb]

I was only working in the folder before the boreal moss folder. I moved but now it doesn't do anything (this is trying to run arrays 1-10).

Job ID: 76824
Array Job ID: 76824_10
Cluster: mp2
User/Group: sarahi/def-ilafores
State: FAILED (exit code 1)
Nodes: 1
Cores per node: 48
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:00:00 core-walltime
Job Wall-clock time: 00:00:00
Memory Utilized: 4.00 EB
Memory Efficiency: 0.00% of 251.00 GB

[jflucier]

I need path to working dir

what do the slurm log say?

Seem to fail right at start

[sarahisherb]

Slurm logs were not even made, logs folder is empty.

pwd is /nfs3_ib/ip29-ib/ip29/ilafores_group/analysis/20220825_boreal_moss/mag_run2

[jflucier]

you need to put full path when you generate script:

bash /project/def-ilafores/common/ILL_pipelines/generateslurm_assembly_bin_refinement.metawrap.sh \
--sample_tsv /nfs3_ib/ip29-ib/ip29/ilafores_group/analysis/20220825_boreal_moss/samples.tsv \
--out /nfs3_ib/ip29-ib/ip29/ilafores_group/analysis/20220825_boreal_moss/mag_run2 \
--refinement --refinement_min_compl 30 --refinement_max_cont 10

[sarahisherb]

Just sent off arrays 1-10, I think it works! The last thing I had to change was to move files from the binning folder in the denovo assembly path to the new path in mag_2 folder. Thanks for your help :)