search

merenlab / anvio

An analysis and visualization platform for 'omics data
http://merenlab.org/software/anvio
GNU General Public License v3.0
426 stars 145 forks source link

Running anvio through snakemake, no errors but the job submission stops running #2333

Closed UniAlberta closed 3 weeks ago

UniAlberta commented 3 weeks ago

Hi,

Im running anvio-8 to do assembly and binning. I can successfully submit the job script with no errors but it fails after a few seconds. There is no error in the output file so Im really confused about it. Could you please give me some guidance?

meren commented 3 weeks ago

Are there any files in the 00_LOGS directory?

UniAlberta commented 3 weeks ago

No, nothing is created.

On Mon, Aug 19, 2024, 23:38 A. Murat Eren (Meren) @.***> wrote:

Are there any files in the 00_LOGS directory?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298006464, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3XVBRE343WCFZROTH3ZSLI3VAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGAYDMNBWGQ . You are receiving this because you authored the thread.Message ID: @.***>

UniAlberta commented 3 weeks ago

The folder 00_LOGS is created but empty.

On Mon, Aug 19, 2024, 23:44 Nafsa Khazaei @.***> wrote:

No, nothing is created.

On Mon, Aug 19, 2024, 23:38 A. Murat Eren (Meren) < @.***> wrote:

Are there any files in the 00_LOGS directory?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298006464, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3XVBRE343WCFZROTH3ZSLI3VAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGAYDMNBWGQ . You are receiving this because you authored the thread.Message ID: @.***>

meren commented 3 weeks ago

This means the problem is occurring before anvi'o processes could actually start running. Can you please run the workflow in an interactive shell rather than sending the command to the queue to see the error messages?

UniAlberta commented 3 weeks ago

I'm running in an interactive shell. The output file has no error messages, but has warning messages.

On Tue, Aug 20, 2024, 02:00 A. Murat Eren (Meren) @.***> wrote:

This means the problem is occurring before anvi'o processes could actually start running. Can you please run the workflow in an interactive shell rather than sending the command to the queue to see the error messages?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298217891, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3V4ZDEPHQG6TM3NLLLZSLZRDAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGIYTOOBZGE . You are receiving this because you authored the thread.Message ID: @.***>

UniAlberta commented 3 weeks ago

Here it is.

On Tue, Aug 20, 2024 at 5:23 AM Nafsa Khazaei @.***> wrote:

I'm running in an interactive shell. The output file has no error messages, but has warning messages.

On Tue, Aug 20, 2024, 02:00 A. Murat Eren (Meren) < @.***> wrote:

This means the problem is occurring before anvi'o processes could actually start running. Can you please run the workflow in an interactive shell rather than sending the command to the queue to see the error messages?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298217891, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3V4ZDEPHQG6TM3NLLLZSLZRDAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGIYTOOBZGE . You are receiving this because you authored the thread.Message ID: @.***>

meren commented 3 weeks ago

Your message says here it is but I don't see anything. Am I fired?

UniAlberta commented 3 weeks ago

Oh no, I wrote here it is in your reply as I attached the output file for you.

On Tue, Aug 20, 2024, 06:51 A. Murat Eren (Meren) @.***> wrote:

Your message says here it is but I don't see anything. Am I fired?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298784071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3XJU2ZGAIF3ES7TWX3ZSM3TPAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYG44DIMBXGE . You are receiving this because you authored the thread.Message ID: @.***>

UniAlberta commented 3 weeks ago

Are you able to see the attached OUT file? If not, I copied the content here.

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

WARNING

If you publish results from this workflow, please do not forget to cite snakemake (doi:10.1093/bioinformatics/bts480)

HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None) HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None) HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None) HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None)

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None)

WARNING

We are initiating parameters for the metagenomics workflow

HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None) HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None)

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

We are initiating parameters for the metagenomics workflow

HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None) HMM profiles .................................: 9 sources have been loaded: Ribosomal_RNA_16S (3 genes, domain: None), Ribosomal_RNA_28S (1 genes, domain: None), Ribosomal_RNA_18S (1 genes, domain: None), Protista_83 (83 genes, domain: eukarya), Ribosomal_RNA_23S (2 genes, domain: None), Bacteria_71 (71 genes, domain: bacteria), Archaea_76 (76 genes, domain: archaea), Ribosomal_RNA_5S (5 genes, domain: None), Ribosomal_RNA_12S (1 genes, domain: None)

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

WARNING

OK, SO THIS IS SERIOUS, AND WHEN THINGS ARE SERIOUS THEN WE USE CAPS. WE SEE THAT YOU ARE USING --additional-params AND THAT'S GREAT, BUT WE WANT TO REMIND YOU THAT ANYTHING THAT FOLLOWS --additional-params WILL BE CONSIDERED AS A snakemake PARAM THAT IS TRANSFERRED TO snakemake DIRECTLY. So make sure that these don't include anything that you didn't mean to include as an additional param: --cores, 12, --keep-going, --rerun-incomplete, --unlock.

WARNING

We are initiating parameters for the metagenomics workflow

/gpfs/fs3c/aafc/aafc_shp/can0002/miniconda3/envs/anvio-8/lib/python3.10/site-packages/anvio/workflows/metagenomics/Snakefile:1111: SyntaxWarning: "is not" with a literal. Did you mean "!="? params:

WARNING

We are initiating parameters for the metagenomics workflow

/gpfs/fs3c/aafc/aafc_shp/can0002/miniconda3/envs/anvio-8/lib/python3.10/site-packages/anvio/workflows/metagenomics/Snakefile:1111: SyntaxWarning: "is not" with a literal. Did you mean "!="? params:

WARNING

We are initiating parameters for the metagenomics workflow

/gpfs/fs3c/aafc/aafc_shp/can0002/miniconda3/envs/anvio-8/lib/python3.10/site-packages/anvio/workflows/metagenomics/Snakefile:1111: SyntaxWarning: "is not" with a literal. Did you mean "!="? params:

WARNING

We are initiating parameters for the metagenomics workflow

/gpfs/fs3c/aafc/aafc_shp/can0002/miniconda3/envs/anvio-8/lib/python3.10/site-packages/anvio/workflows/metagenomics/Snakefile:1111: SyntaxWarning: "is not" with a literal. Did you mean "!="? params:

WARNING

We are initiating parameters for the metagenomics workflow

Shell programs for the workflow

Needed .......................................: gunzip, anvi-script-reformat-fasta, anvi-script-reformat-fasta, anvi-gen-contigs-database, anvi-import-functions, anvi-get-sequences-for-gene-calls, centrifuge, anvi-import-taxonomy-for-genes, anvi-run-hmms, anvi-run-pfams, anvi-run-kegg-kofams, anvi-run-ncbi-cogs, anvi-run-scg-taxonomy, anvi-scan-trnas, anvi-get-sequences-for-gene-calls, iu-gen-configs, iu-filter-quality-minoche, gzip, bowtie2-build, bowtie2, samtools, anvi-init-bam, anvi-profile, echo, anvi-import-collection, anvi-script-add-default-collection, anvi-summarize, anvi-split, mv, krakenuniq, krakenuniq-mpa-report, anvi-import-taxonomy-for-layers, anvi-cluster-contigs Missing ......................................: centrifuge, krakenuniq, krakenuniq-mpa-report

Shell programs for the workflow

Needed .......................................: gunzip, anvi-script-reformat-fasta, anvi-script-reformat-fasta, anvi-gen-contigs-database, anvi-import-functions, anvi-get-sequences-for-gene-calls, centrifuge, anvi-import-taxonomy-for-genes, anvi-run-hmms, anvi-run-pfams, anvi-run-kegg-kofams, anvi-run-ncbi-cogs, anvi-run-scg-taxonomy, anvi-scan-trnas, anvi-get-sequences-for-gene-calls, iu-gen-configs, iu-filter-quality-minoche, gzip, bowtie2-build, bowtie2, samtools, anvi-init-bam, anvi-profile, echo, anvi-import-collection, anvi-script-add-default-collection, anvi-summarize, anvi-split, mv, krakenuniq, krakenuniq-mpa-report, anvi-import-taxonomy-for-layers, anvi-cluster-contigs Missing ......................................: centrifuge, krakenuniq, krakenuniq-mpa-report

/gpfs/fs3c/aafc/aafc_shp/can0002/miniconda3/envs/anvio-8/lib/python3.10/site-packages/anvio/workflows/metagenomics/Snakefile:1111: SyntaxWarning: "is not" with a literal. Did you mean "!="? params:

WARNING

We are initiating parameters for the metagenomics workflow

Shell programs for the workflow

Needed .......................................: gunzip, anvi-script-reformat-fasta, anvi-script-reformat-fasta, anvi-gen-contigs-database, anvi-import-functions, anvi-get-sequences-for-gene-calls, centrifuge, anvi-import-taxonomy-for-genes, anvi-run-hmms, anvi-run-pfams, anvi-run-kegg-kofams, anvi-run-ncbi-cogs, anvi-run-scg-taxonomy, anvi-scan-trnas, anvi-get-sequences-for-gene-calls, iu-gen-configs, iu-filter-quality-minoche, gzip, bowtie2-build, bowtie2, samtools, anvi-init-bam, anvi-profile, echo, anvi-import-collection, anvi-script-add-default-collection, anvi-summarize, anvi-split, mv, krakenuniq, krakenuniq-mpa-report, anvi-import-taxonomy-for-layers, anvi-cluster-contigs Missing ......................................: centrifuge, krakenuniq, krakenuniq-mpa-report

On Tue, Aug 20, 2024 at 9:13 AM Nafsa Khazaei @.***> wrote:

Oh no, I wrote here it is in your reply as I attached the output file for you.

On Tue, Aug 20, 2024, 06:51 A. Murat Eren (Meren) < @.***> wrote:

Your message says here it is but I don't see anything. Am I fired?

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2298784071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3XJU2ZGAIF3ES7TWX3ZSM3TPAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYG44DIMBXGE . You are receiving this because you authored the thread.Message ID: @.***>

meren commented 3 weeks ago

I see --unlock in your additional parameters. It is simply asking snakemake to remove a stale lock (if exists), and quit. Please read the snakemake help from more details (and perhaps remove that parameter and try again after).

UniAlberta commented 3 weeks ago

Hi, I removed the --unlock argument from my script. There is an error in the output file: Shell programs for the workflow

Needed .......................................: gunzip, anvi-script-reformat-fasta, anvi-script-reformat-fasta, anvi-gen-contigs-database, anvi-import-functions, anvi-get-sequences-for-gene-calls, centrifuge, anvi-import-taxonomy-for-genes, anvi-run-hmms, anvi-run-pfams, anvi-run-kegg-kofams, anvi-run-ncbi-cogs, anvi-run-scg-taxonomy, anvi-scan-trnas, anvi-get-sequences-for-gene-calls, iu-gen-configs, iu-filter-quality-minoche, gzip, bowtie2-build, bowtie2, samtools, anvi-init-bam, anvi-profile, echo, anvi-import-collection, anvi-script-add-default-collection, anvi-summarize, anvi-split, mv, krakenuniq, krakenuniq-mpa-report, anvi-import-taxonomy-for-layers, anvi-cluster-contigs Missing ......................................: centrifuge, krakenuniq, krakenuniq-mpa-report

Building DAG of jobs... LockException: Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following directory: ~/nak000/anvio If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with the --unlock argument.

However, I see the folders such as 00_LOGS, 02_FASTA, 03_CONTIGS and 04_MAPPING have been created with files in them and the workflow is running. Is that fine?

On Wed, Aug 21, 2024 at 2:23 AM A. Murat Eren (Meren) < @.***> wrote:

I see --unlock in your additional parameters. It is simply asking snakemake to remove a stale lock (if exists), and quit. Please read the snakemake help from more details (and perhaps remove that parameter and try again after).

— Reply to this email directly, view it on GitHub https://github.com/merenlab/anvio/issues/2333#issuecomment-2301452545, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZVXR3Q2IZSXT5INA6LT3B3ZSRE6JAVCNFSM6AAAAABMYRY6VSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBRGQ2TENJUGU . You are receiving this because you authored the thread.Message ID: @.***>

meren commented 3 weeks ago

No this is not fine :( I'm afraid you need to talk to a local bioinformatics person. This needs some interactive debugging to figure out what is happening with your snakemake. Sorry.