search

jtamames / SqueezeMeta

A complete pipeline for metagenomic analysis
GNU General Public License v3.0
357 stars 78 forks source link

re-start with added options #633

Closed spongebob22 closed 1 year ago

spongebob22 commented 1 year ago

Hi there,

I am running v1.6.1 on a HPC.

Last time i used a previous version of SqueezeMeta the default binners were maxbin and metabat. I updated to the new version i did not noticed the defaults are now concoct, metabat. I am in the middle of a run -- is it possible for me to use "restart" with added binner options so that all three method are used. If so would i run that as so:

SqueezeMeta.pl -p --restart -step 14 -binners maxbin, metabat, concoct

Thanks,

jtamames commented 1 year ago

Hello

Yes, it is possible, but not in that way. Edit the SqeezeMeta_conf.pl in the project directory, and change:

$binners         = "concoct,metabat2";

by:

$binners         = "concoct,metabat2,maxbin";

Then restart with:

SqueezeMeta.pl -p --restart -step 14 --force_overwrite

That will do it.

Best,

J

On 28/2/23 1:14, spongebob22 wrote:

Hi there,

I am running v1.6.1 on a HPC.

Last time i used a previous version of SqueezeMeta the default binners were maxbin and metabat. I updated to the new version i did not noticed the defaults are now concoct, metabat. I am in the middle of a run -- is it possible for me to use "restart" with added binner options so that all three method are used. If so would i run that as so:

SqueezeMeta.pl -p --restart -step 14 -binners maxbin, metabat, concoct

Thanks,

— Reply to this email directly, view it on GitHub https://github.com/jtamames/SqueezeMeta/issues/633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIUX7NSCZV46KMDUIGGUYDWZU7M5ANCNFSM6AAAAAAVKBYBQQ. You are receiving this because you are subscribed to this thread.Message ID: @.***>

spongebob22 commented 1 year ago

thank you for the explanation.

Sorry if this is obvious but will "--force_overwrite" just overwrite step 14 then carry on with the pipeline? Or will it start every step over from step1?

the pipeline has already run steps 1-14. I stopped it mid way through step 14.

jtamames commented 1 year ago

It will rewrite everything from step 14.

Best,

J

On 28/2/23 17:39, spongebob22 wrote:

thank you for the explanation.

Sorry if this is obvious but will "--force_overwrite" just overwrite step 14 then carry on with the pipeline? Or will it start every step over from step1?

the pipeline has already run steps 1-14. I stopped it mid way through step 14.

— Reply to this email directly, view it on GitHub https://github.com/jtamames/SqueezeMeta/issues/633#issuecomment-1448497989, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIUX7OQMBXA53LXQTMI2XDWZYS3VANCNFSM6AAAAAAVKBYBQQ. You are receiving this because you commented.Message ID: @.***>

spongebob22 commented 1 year ago

Thanks for clarifying. It restarted and seems to be running but its been stuck on CONCOCT for about 60hours. Not sure if thats normal

The log file looks like this:

GENE TABLE CREATED: /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/results/13.seds_squeezeM_coass_021723.orftable
[2 days, 22 hours, 50 minutes, 43 seconds]: STEP14 -> 14.runbinning.pl
RUNNING concoct
  Working for seds_squeezeM_coass_021723.sed3.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed3.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/
user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_0
21723.sed3.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed3.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/Squeez
eMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed3.sorted.bam > /dev/null 2>&1
  Working for seds_squeezeM_coass_021723.sed1.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed1.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/rsimi
ste/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_021723
.sed1.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed1.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed1.sorted.bam > /dev/null 2>&1
  Working for seds_squeezeM_coass_021723.sed2.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed2.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_021723.sed2.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed2.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed2.sorted.bam > /dev/null 2>&1

  Cutting contigs in pieces!: python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/scripts/cut_up_fasta.py /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/results/01.seds_squeezeM_coass_021723.fasta -c 10000 -o 0 --merge_last -b /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.contigs.bed > /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.choppedcontigs.fasta
  Creating abundance table: PATH=/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin:$PATH python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/scripts/concoct_coverage_table.py /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.contigs.bed /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/*.sorted.bam > /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/coverage_table.tsv
  Running concoct: OMP_THREAD_LIMIT=30 python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/bin/concoct --composition_file /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.choppedcontigs.fasta --coverage_file /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/coverage_table.tsv --threads 30  -b /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/concoct_int/ > /dev/null 2>&1

RUNNING concoct
  Working for seds_squeezeM_coass_021723.sed3.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed3.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_021723.sed3.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed3.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed3.sorted.bam > /dev/null 2>&1
  Working for seds_squeezeM_coass_021723.sed1.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed1.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_021723.sed1.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed1.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed1.sorted.bam > /dev/null 2>&1
  Working for seds_squeezeM_coass_021723.sed2.sam
  Creating sorted BAM for seds_squeezeM_coass_021723.sed2.sam: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools sort /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/data/sam/seds_squeezeM_coass_021723.sed2.sam -o /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed2.sorted.bam -@30 > /dev/null 2>&1
  Indexing sorted BAM: LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/lib /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/samtools index /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.sed2.sorted.bam > /dev/null 2>&1

  Cutting contigs in pieces!: python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/scripts/cut_up_fasta.py /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/results/01.seds_squeezeM_coass_021723.fasta -c 10000 -o 0 --merge_last -b /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.contigs.bed > /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.choppedcontigs.fasta
  Creating abundance table: PATH=/arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin:$PATH python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/scripts/concoct_coverage_table.py /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.contigs.bed /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/*.sorted.bam > /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/coverage_table.tsv
  Running concoct: OMP_THREAD_LIMIT=30 python3 /arc/home/user/squeezemeta_v161/envs/SqueezeMeta/SqueezeMeta/bin/CONCOCT-1.1.0/bin/concoct --composition_file /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/temp/seds_squeezeM_coass_021723.choppedcontigs.fasta --coverage_file /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/coverage_table.tsv --threads 30  -b /scratch/st-sacrowe-1/seds_lake_squeezem_021723/seds_squeezeM_coass_021723/intermediate/binners/concoct/concoct_int/ > /dev/null 2>&1

Any thoughts?

spongebob22 commented 1 year ago

The job died after 90 hours with not change.

I tried again but did the restarted the program earlier "SqueezeMeta.pl -p --restart -step 13 --force_overwrite"

but it got stuck trying to run concoct for 120 hours then exceeded the wall time.

Not sure what the issue is.....

fpusan commented 1 year ago

Is this a big coassembly of many samples? Maybe CONCOCT is just taking that much to finish

fpusan commented 1 year ago

Closing due to lack of activity, feel free to reopen