search

metagenome-atlas / atlas

ATLAS - Three commands to start analyzing your metagenome data
https://metagenome-atlas.github.io/
BSD 3-Clause "New" or "Revised" License
368 stars 97 forks source link

snakemake issue with atlas assemble #2

Closed mdjlynch closed 7 years ago

mdjlynch commented 7 years ago

Hello. Having an issue with atlas assemble on Ubuntu 16.04. Dependencies have been installed and using atlas_env.

atlas assemble mgm4637809.3.yaml ... subprocess.CalledProcessError: Command 'snakemake -s... returned non-zero exit status 1

Looks like a very useful pipeline. Any help is appreciated.

Thanks.

raw937 commented 7 years ago

Dr. Lynch thank you so much for testing/using ATLAS. Tell Josh N - hello from me as well. It appears to be a python error. Is python version 3.5 currently installed?

Cheers Rick

mdjlynch commented 7 years ago

I will. I've been wanting to try ATLAS ever since seeing the preprint.

Yes, I have python 3.5 installed: Python 3.5.3 |Continuum Analytics, Inc.| (default, Mar 6 2017, 11:58:13) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux

Error happens directly after: mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cog/mgm4637809.3_hits.tsv

Thanks.

brwnj commented 7 years ago

Can you re-run the command and post the full output? The subprocess error is from atlas assemble invoking snakemake and doesn't show the actual issue which would have occurred slightly upstream.

mdjlynch commented 7 years ago

Sure. Here's the traceback of the error:

mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cog/mgm4637809.3_hits.tsv
Traceback (most recent call last):
  File "/home/mdjlynch/miniconda3/envs/atlas_env/bin/atlas", line 11, in <module>
    sys.exit(cli())
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/atlas/atlas.py", line 300, in run_assemble
    assemble(os.path.realpath(config), jobs, out_dir, dryrun, snakemake_args)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/atlas/workflows.py", line 28, in assemble
    check_call(cmd, shell=True)
  File "/home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/subprocess.py", line 271, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'snakemake -s /home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/atlas/Snakefile -d /media/Data/metagenom-1-data/metagenomes/NEON/assemblies/mgm4637809.3 -p -j 4 --rerun-incomplete --configfile '/media/Data/metagenom-1-data/metagenomes/NEON/assemblies/mgm4637809.3/mgm4637809.3.yaml' --nolock --config workflow=complete  --' returned non-zero exit status 1

Thanks for your help.

brwnj commented 7 years ago

There is likely a failure just above that and may be more obvious if you try to run:

snakemake -s /home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/atlas/Snakefile -d /media/Data/metagenom-1-data/metagenomes/NEON/assemblies/mgm4637809.3 -p -j 4 --rerun-incomplete --configfile '/media/Data/metagenom-1-data/metagenomes/NEON/assemblies/mgm4637809.3/mgm4637809.3.yaml' --nolock --config workflow=complete
mdjlynch commented 7 years ago

Okay, there was a failure above that off screen:

MissingInputException in line 196 of /home/mdjlynch/miniconda3/envs/atlas_env/lib/python3.5/site-packages/atlas/Snakefile:
Missing input files for rule all:
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_taxonomy_species.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_RefSeq.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/eggnog/mgm4637809.3_assignments.tsv
mgm4637809.3/quality_control/decontamination/mgm4637809.3_PhiX.fastq.gz
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_CAZy_Family_taxonomy_class.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/enzyme/mgm4637809.3_assignments.tsv
mgm4637809.3/quality_control/quality_filter/mgm4637809.3_pe.fastq.gz
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/genomic_bins/mgm4637809.3.summary
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_ENZYME_taxonomy_species.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/stats/prefilter_contig_stats.txt
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_COG_taxonomy_order.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_taxonomy_class.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_ENZYME.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_RefSeq_taxonomy_phylum.tsv
mgm4637809.3/quality_control/normalization_k21_t100/mgm4637809.3_pe.fastq.gz
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cog/mgm4637809.3_assignments.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_CAZy_Family_taxonomy_phylum.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cog/mgm4637809.3_hits.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_taxonomy_order.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_COG.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_ENZYME_taxonomy_order.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_ENZYME_taxonomy_phylum.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_COG_taxonomy_class.tsv
mgm4637809.3/quality_control/decontamination/mgm4637809.3_refstats.txt
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/stats/final_contig_stats.txt
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/mgm4637809.3_contigs.fasta
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cazy/mgm4637809.3_assignments.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_ENZYME_taxonomy_class.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_CAZy_Family_taxonomy_order.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_RefSeq_taxonomy_species.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/enzyme/mgm4637809.3_hits.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_CAZy_Family_taxonomy_species.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_KO.tsv
mgm4637809.3/logs/mgm4637809.3_quality_filtering_stats.txt
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/refseq/mgm4637809.3_assignments.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/cazy/mgm4637809.3_hits.tsv
mgm4637809.3/quality_control/quality_filter/mgm4637809.3_se.fastq.gz
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/orfs/mgm4637809.3.faa
mgm4637809.3/quality_control/decontamination/mgm4637809.3_rRNA.fastq.gz
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_RefSeq_taxonomy_class.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/mgm4637809.3_merged_assignments.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/eggnog/mgm4637809.3_hits.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_RefSeq_taxonomy_order.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_CAZy.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_COG_taxonomy_phylum.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_COG_taxonomy_species.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/annotation/refseq/mgm4637809.3_hits.tsv
mgm4637809.3/megahit_21_121_20_normalization_k21_t100/count_tables/mgm4637809.3_taxonomy_phylum.tsv
brwnj commented 7 years ago

This suggests that from where the command was invoked, the workflow infrastructure was unable to locate input files. In your configuration file, if your sample file paths are defined as relative paths (they do not start with '/'), then you'll have to invoke the command from the directory that satisfies the relative paths.

Do you have absolute paths defined for mgm4637809.3 now?

Thanks for taking the time to send this feedback!

mdjlynch commented 7 years ago

Yes, I confirmed that I have been using absolute paths. I have also tried running the assemble command from within the same directory with the same result. I will try reinstalling atlas and running the analysis from my home directory.

mdjlynch commented 7 years ago

Okay, the problem appears to be the input fastq files. Not sure what the issue is yet, but the atlas pipeline is running fine on two other projects.

Thanks for your time.