search

marbl / MetaCompass

MetaCompass: Reference-guided Assembly of Metagenomes
https://github.com/marbl/MetaCompass/wiki
Other
38 stars 11 forks source link

Both tutorial examples are failing #3

Closed nick-youngblut closed 6 years ago

nick-youngblut commented 6 years ago

I'm trying to run the MetaCompass examples listed in the README, and I've run into errors for both.

Install

conda create -n metacompass bioconda::snakemake bioconda::blast bioconda::meryl bioconda::samtools bioconda::megahit bioconda::pilon
source activate metacompass

git clone https://github.com/marbl/MetaCompass.git
cd MetaCompass
./install.sh

Example 1

Command

./go_metacompass.py -r tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -o example1_output -m 3 -t 4

Output

$ ./go_metacompass.py -r tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -o example1_output -m 3 -t 4 --clobber
confirming file containing reference genomes exists..
[OK]
checking for dependencies (Bowtie2, Blast, kmermask, Snakemake, etc)
Bowtie2--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/blastn
Blast+--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/kmer-mask
kmer-mask--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/snakemake
Snakemake--->[OK]
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    1   assemble_unmapped
    1   bam_sort
    1   bowtie2_map
    1   build_contigs
    1   create_tsv
    1   join_contigs
    1   merge_reads
    1   pilon_contigs
    1   pilon_map
    1   sam_to_bam
    11
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
    merge_reads
Selected jobs (1):
    merge_reads
Resources after job selection: {'_cores': 3, '_nodes': 9223372036854775806}

Job 10: ---merge fastq reads
Reason: Missing output files: example1_output/thao2000.merged.fq

Building DAG of jobs...
Using shell: /bin/bash
Job counts:
    count   jobs
    1   merge_reads
    1
Complete log: /ebio/abt3_projects/software/dev/tmp/MetaCompass/.snakemake/log/2018-05-11T164650.403818.snakemake.log
Releasing 1 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
Finished job 10.
1 of 11 steps (9%) done
Resources before job selection: {'_cores': 4, '_nodes': 9223372036854775807}
Ready jobs (1):
    bowtie2_map
Selected jobs (1):
    bowtie2_map
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

Job 9: ---Build index .
Reason: Missing output files: example1_output/thao2000.0.assembly.out/thao2000.sam; Input files updated by another job: example1_output/thao2000.merged.fq

Full Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 1324, in run_wrapper
    singularity_args, use_singularity, None)
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py", line 88, in __rule_bowtie2_map
    rule sam_to_bam:
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/shell.py", line 110, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  bowtie2-build -o 3 --threads 4 -q tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta example1_output/thao2000.0.assembly.out/thao2000.index 1>> example1_output/thao2000.0.assembly.out/thao2000.index 2>&1;bowtie2 -a --end-to-end --sensitive --no-unal -p 4 -x example1_output/thao2000.0.assembly.out/thao2000.index -q -U example1_output/thao2000.merged.fq -S example1_output/thao2000.0.assembly.out/thao2000.sam.all > example1_output/thao2000.0.bowtie2map.log 2>&1; /ebio/abt3_projects/software/dev/tmp/MetaCompass/bin/best_strata.py example1_output/thao2000.0.assembly.out/thao2000.sam.all example1_output/thao2000.0.assembly.out/thao2000.sam; rm example1_output/thao2000.0.assembly.out/thao2000.sam.all ' returned non-zero exit status 1.

Error in rule bowtie2_map:
    jobid: 9
    output: example1_output/thao2000.0.assembly.out/thao2000.index, example1_output/thao2000.0.assembly.out/thao2000.index, example1_output/thao2000.0.assembly.out/thao2000.sam
    log: example1_output/thao2000.0.bowtie2map.log

Full Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 1324, in run_wrapper
    singularity_args, use_singularity, None)
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py", line 88, in __rule_bowtie2_map
    rule sam_to_bam:
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/shell.py", line 110, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ' set -euo pipefail;  bowtie2-build -o 3 --threads 4 -q tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta example1_output/thao2000.0.assembly.out/thao2000.index 1>> example1_output/thao2000.0.assembly.out/thao2000.index 2>&1;bowtie2 -a --end-to-end --sensitive --no-unal -p 4 -x example1_output/thao2000.0.assembly.out/thao2000.index -q -U example1_output/thao2000.merged.fq -S example1_output/thao2000.0.assembly.out/thao2000.sam.all > example1_output/thao2000.0.bowtie2map.log 2>&1; /ebio/abt3_projects/software/dev/tmp/MetaCompass/bin/best_strata.py example1_output/thao2000.0.assembly.out/thao2000.sam.all example1_output/thao2000.0.assembly.out/thao2000.sam; rm example1_output/thao2000.0.assembly.out/thao2000.sam.all ' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 378, in _callback
    raise ex
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/executors.py", line 1336, in run_wrapper
    show_traceback=True))
snakemake.exceptions.RuleException: CalledProcessError in line 50 of /ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py:
Command ' set -euo pipefail;  bowtie2-build -o 3 --threads 4 -q tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta example1_output/thao2000.0.assembly.out/thao2000.index 1>> example1_output/thao2000.0.assembly.out/thao2000.index 2>&1;bowtie2 -a --end-to-end --sensitive --no-unal -p 4 -x example1_output/thao2000.0.assembly.out/thao2000.index -q -U example1_output/thao2000.merged.fq -S example1_output/thao2000.0.assembly.out/thao2000.sam.all > example1_output/thao2000.0.bowtie2map.log 2>&1; /ebio/abt3_projects/software/dev/tmp/MetaCompass/bin/best_strata.py example1_output/thao2000.0.assembly.out/thao2000.sam.all example1_output/thao2000.0.assembly.out/thao2000.sam; rm example1_output/thao2000.0.assembly.out/thao2000.sam.all ' returned non-zero exit status 1.
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py", line 50, in __rule_bowtie2_map

RuleException:
CalledProcessError in line 50 of /ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py:
Command ' set -euo pipefail;  bowtie2-build -o 3 --threads 4 -q tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta example1_output/thao2000.0.assembly.out/thao2000.index 1>> example1_output/thao2000.0.assembly.out/thao2000.index 2>&1;bowtie2 -a --end-to-end --sensitive --no-unal -p 4 -x example1_output/thao2000.0.assembly.out/thao2000.index -q -U example1_output/thao2000.merged.fq -S example1_output/thao2000.0.assembly.out/thao2000.sam.all > example1_output/thao2000.0.bowtie2map.log 2>&1; /ebio/abt3_projects/software/dev/tmp/MetaCompass/bin/best_strata.py example1_output/thao2000.0.assembly.out/thao2000.sam.all example1_output/thao2000.0.assembly.out/thao2000.sam; rm example1_output/thao2000.0.assembly.out/thao2000.sam.all ' returned non-zero exit status 1.
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.ref.py", line 50, in __rule_bowtie2_map
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing output files of failed job bowtie2_map since they might be corrupted:
example1_output/thao2000.0.assembly.out/thao2000.index, example1_output/thao2000.0.assembly.out/thao2000.index
Releasing 4 _cores (now 4).
Releasing 1 _nodes (now 9223372036854775807).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /ebio/abt3_projects/software/dev/tmp/MetaCompass/.snakemake/log/2018-05-11T164647.579389.snakemake.log
unlocking
removing lock
removing lock
removed all locks
ERROR: snakemake command failed; exiting..

Example 2

Command

wget ftp://public-ftp.hmpdacc.org/Illumina/posterior_fornix/SRS044742.tar.bz2
tar xvjf SRS044742.tar.bz2
./go_metacompass.py -P SRS044742/SRS044742.denovo_duplicates_marked.trimmed.1.fastq,SRS044742/SRS044742.denovo_duplicates_marked.trimmed.2.fastq -U SRS044742/SRS044742.denovo_duplicates_marked.trimmed.singleton.fastq -o example2_output

Output

$ ./go_metacompass.py -P SRS044742/SRS044742.denovo_duplicates_marked.trimmed.1.fastq,SRS044742/SRS044742.denovo_duplicates_marked.trimmed.2.fastq -U SRS044742/SRS044742.denovo_duplicates_marked.trimmed.singleton.fastq -o example2_output
checking for dependencies (Bowtie2, Blast, kmermask, Snakemake, etc)
Bowtie2--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/blastn
Blast+--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/kmer-mask
kmer-mask--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/snakemake
Snakemake--->[OK]
Full Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='output', start=(78, 4), end=(78, 10), line="    output:expand('{prefix}/{sample}.fasta',prefix=config['prefix'],sample=config['sample'])\n")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='message', start=(79, 4), end=(79, 11), line='    message: """---Converting fastq to fasta."""\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='shell', start=(80, 4), end=(80, 9), line='    shell : "perl %s/bin/fq2fa.pl -i {input} -o {output}"%(config["mcdir"])\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='log', start=(81, 4), end=(81, 7), line="    log:'%s/%s.%s.fastq2fasta.log'%(config['prefix'],config['sample'],config['iter'])\n")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/__init__.py", line 382, in snakemake
    print_compilation=print_compilation)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/workflow.py", line 652, in include
    rulecount=self._rulecount)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 766, in parse
    for t, orig_token in automaton.consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 710, in python
    for t in self.subautomaton(token.string).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 644, in block_content
    "rule {}.".format(self.rulename), token)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.6/site-packages/snakemake/parser.py", line 117, in error
    (self.snakefile.path, lineno(token), None, None))
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.py", line 81
SyntaxError: No rule keywords allowed after run/shell/script/wrapper/cwl in rule fastq2fasta.

SyntaxError in line 81 of /ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.py:
No rule keywords allowed after run/shell/script/wrapper/cwl in rule fastq2fasta. (metacompass.iter0.py, line 81)
ERROR: snakemake command failed; exiting..
touch: cannot touch 'example2_output/SRS044742.0.assembly.out/run.fail': No such file or directory

Conda environment

# packages in environment at /ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass:
#
# Name                    Version                   Build  Channel
aioeasywebdav             2.2.0                    py36_0    conda-forge
aiohttp                   3.2.1                    py36_0    conda-forge
appdirs                   1.4.3                      py_0    conda-forge
asn1crypto                0.24.0                   py36_0    conda-forge
async-timeout             2.0.1                    py36_0    conda-forge
attrs                     18.1.0                     py_0    conda-forge
bcrypt                    3.1.4            py36h621fe67_0
blast                     2.7.1               boost1.64_3    bioconda
boost                     1.64.0                   py36_4    conda-forge
boost-cpp                 1.64.0                        1    conda-forge
boto3                     1.7.19                     py_0    conda-forge
botocore                  1.10.19                    py_0    conda-forge
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
cachetools                2.0.1                      py_0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
cffi                      1.11.5                   py36_0    conda-forge
chardet                   3.0.4                    py36_0    conda-forge
configargparse            0.12.0                   py36_0    conda-forge
cryptography              2.2.1                    py36_0    conda-forge
curl                      7.59.0                        1    conda-forge
datrie                    0.7.1                    py36_0
docutils                  0.14                     py36_0    conda-forge
dropbox                   8.8.1                      py_0    conda-forge
filechunkio               1.8                      py36_1    conda-forge
ftputil                   3.4                        py_0    conda-forge
gmp                       6.1.2                         0    conda-forge
gnutls                    3.5.17                        0    conda-forge
google-auth               1.4.1                      py_0    conda-forge
google-auth-httplib2      0.0.2                    py36_0    conda-forge
google-cloud-core         0.24.1                   py36_0    conda-forge
google-cloud-storage      1.1.1                    py36_0    conda-forge
google-resumable-media    0.0.2                    py36_0    conda-forge
googleapis-common-protos  1.5.3                    py36_0    conda-forge
httplib2                  0.11.1                   py36_0    conda-forge
icu                       58.2                          0    conda-forge
idna                      2.6                      py36_1    conda-forge
idna_ssl                  1.0.0                         0    conda-forge
intel-openmp              2018.0.0                      8
jmespath                  0.9.3                    py36_0    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                0.8                           0    bioconda
libffi                    3.2.1                         3    conda-forge
libgcc                    7.2.0                h69d50b8_2
libgcc-ng                 7.2.0                hdf63c60_3
libgfortran-ng            7.2.0                hdf63c60_3
libidn11                  1.33                          0    conda-forge
libprotobuf               3.5.2                         0    conda-forge
libssh2                   1.8.0                         2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3
megahit                   1.1.2                    py36_1    bioconda
meryl                     2013                          0    bioconda
mkl                       2018.0.2                      1
mkl_fft                   1.0.2                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
multidict                 4.3.1                    py36_0    conda-forge
ncurses                   5.9                          10    conda-forge
nettle                    3.3                           0    conda-forge
numpy                     1.14.3           py36h14a74c5_0
numpy-base                1.14.3           py36hdbf6ddf_0
openjdk                   8.0.121                       1
openssl                   1.0.2o                        0    conda-forge
pandas                    0.22.0                   py36_1    conda-forge
paramiko                  2.4.1                    py36_0    conda-forge
pcre                      8.41                          1    conda-forge
perl                      5.22.0.1                      0    conda-forge
perl-app-cpanminus        1.7043               pl5.22.0_0    bioconda
perl-archive-tar          2.18                 pl5.22.0_2    bioconda
perl-carp                 1.38                 pl5.22.0_0    bioconda
perl-compress-raw-bzip2   2.069                         1    bioconda
perl-compress-raw-zlib    2.069                         3    bioconda
perl-data-dumper          2.161                pl5.22.0_0    bioconda
perl-exporter             5.72                 pl5.22.0_0    bioconda
perl-exporter-tiny        0.042                         1    bioconda
perl-extutils-makemaker   7.24                 pl5.22.0_1    bioconda
perl-io-compress          2.069                pl5.22.0_2    bioconda
perl-io-zlib              1.10                          1    bioconda
perl-list-moreutils       0.428                pl5.22.0_0    bioconda
perl-pathtools            3.73                          0    bioconda
perl-scalar-list-utils    1.45                          2    bioconda
perl-test-more            1.001002             pl5.22.0_0    bioconda
perl-threaded             5.22.0              pl5.22.0_12    bioconda
pilon                     1.22                     py36_0    bioconda
pip                       9.0.3                    py36_0    conda-forge
prettytable               0.7.2                    py36_1    conda-forge
protobuf                  3.5.2                    py36_0    conda-forge
psutil                    5.4.5                    py36_0    conda-forge
pyasn1                    0.4.2                      py_0    conda-forge
pyasn1-modules            0.2.1                      py_0    conda-forge
pycparser                 2.18                     py36_0    conda-forge
pynacl                    1.1.2                    py36_0    conda-forge
pyopenssl                 17.5.0                   py36_1    conda-forge
pysftp                    0.2.9                    py36_0    conda-forge
pysocks                   1.6.8                    py36_1    conda-forge
python                    3.6.5                         1    conda-forge
python-dateutil           2.7.2                      py_0    conda-forge
python-irodsclient        0.7.0                      py_0    conda-forge
pytz                      2018.4                     py_0    conda-forge
pyyaml                    3.12                     py36_1    conda-forge
ratelimiter               1.2.0                    py36_0    conda-forge
readline                  7.0                           0    conda-forge
requests                  2.18.4                   py36_1    conda-forge
rsa                       3.4.2                    py36_0    conda-forge
s3transfer                0.1.13                   py36_0    conda-forge
samtools                  1.8                           3    bioconda
setuptools                39.1.0                   py36_0    conda-forge
six                       1.11.0                   py36_1    conda-forge
snakemake                 4.8.1                    py36_0    bioconda
sqlite                    3.20.1                        2    conda-forge
tk                        8.6.7                         0    conda-forge
urllib3                   1.22                     py36_0    conda-forge
wheel                     0.31.0                   py36_0    conda-forge
wrapt                     1.10.11                  py36_0    conda-forge
xmlrunner                 1.7.7                      py_0    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.7                         0    conda-forge
yarl                      1.2.3                    py36_0    conda-forge
zlib                      1.2.11                        0    conda-forge
vcepeda commented 6 years ago

Which version of bowtie2 are you using? MetaCompass requires bowtie2.2.9. If that's not the problem can you share "example1_output/thao2000.0.bowtie2map.log"?

nick-youngblut commented 6 years ago

Whoops, yeah I was accidentally using an old version of bowtie2 not in the conda environment. The Bowtie2--->[OK] line lulled me into thinking that the bowtie2 dependency was OK. Installing bowtie2 fixed the first example error, but not the 2nd:

$ ./go_metacompass.py -P SRS044742/SRS044742.denovo_duplicates_marked.trimmed.1.fastq,SRS044742/SRS044742.denovo_duplicates_marked.trimmed.2.fastq -U SRS044742/SRS044742.denovo_duplicates_marked.trimmed.singleton.fastq -o example2_output
checking for dependencies (Bowtie2, Blast, kmermask, Snakemake, etc)
Bowtie2--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/blastn
Blast+--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/kmer-mask
kmer-mask--->[OK]
/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/bin/snakemake
Snakemake--->[OK]
Full Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='output', start=(78, 4), end=(78, 10), line="    output:expand('{prefix}/{sample}.fasta',prefix=config['prefix'],sample=config['sample'])\n")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='message', start=(79, 4), end=(79, 11), line='    message: """---Converting fastq to fasta."""\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='shell', start=(80, 4), end=(80, 9), line='    shell : "perl %s/bin/fq2fa.pl -i {input} -o {output}"%(config["mcdir"])\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 646, in block_content
    rulename=self.rulename).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 170, in block
    raise StopAutomaton(token)
snakemake.parser.StopAutomaton: TokenInfo(type=1 (NAME), string='log', start=(81, 4), end=(81, 7), line="    log:'%s/%s.%s.fastq2fasta.log'%(config['prefix'],config['sample'],config['iter'])\n")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/__init__.py", line 382, in snakemake
    print_compilation=print_compilation)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/workflow.py", line 652, in include
    rulecount=self._rulecount)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 766, in parse
    for t, orig_token in automaton.consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 710, in python
    for t in self.subautomaton(token.string).consume():
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 105, in consume
    for t, orig in self.state(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 653, in block_content
    for t in self.block(e.token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 179, in block
    for t in self.block_content(token):
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 644, in block_content
    "rule {}.".format(self.rulename), token)
  File "/ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass/lib/python3.5/site-packages/snakemake/parser.py", line 117, in error
    (self.snakefile.path, lineno(token), None, None))
  File "/ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.py", line 81
SyntaxError: No rule keywords allowed after run/shell/script/wrapper/cwl in rule fastq2fasta.

SyntaxError in line 81 of /ebio/abt3_projects/software/dev/tmp/MetaCompass/snakemake/metacompass.iter0.py:
No rule keywords allowed after run/shell/script/wrapper/cwl in rule fastq2fasta. (metacompass.iter0.py, line 81)
ERROR: snakemake command failed; exiting..
touch: cannot touch 'example2_output/SRS044742.0.assembly.out/run.fail': No such file or directory

It appears that there's a legit syntax error in the fastq2fasta rule. I can't tell what the problem is though. I've used snakemake before, but I can't see what the problem is.

My updated conda environment:

# packages in environment at /ebio/abt3_projects/software/dev/miniconda3_dev/envs/metacompass:
#
# Name                    Version                   Build  Channel
aioeasywebdav             2.2.0                    py35_0    conda-forge
aiohttp                   3.2.1                    py35_0    conda-forge
appdirs                   1.4.3                      py_0    conda-forge
asn1crypto                0.24.0                   py35_0    conda-forge
async-timeout             2.0.1                    py35_0    conda-forge
attrs                     18.1.0                     py_0    conda-forge
bcrypt                    3.1.4            py35ha35c455_0
blast                     2.7.1               boost1.64_3    bioconda
boost                     1.64.0                   py35_4    conda-forge
boost-cpp                 1.64.0                        1    conda-forge
boto3                     1.7.19                     py_0    conda-forge
botocore                  1.10.19                    py_0    conda-forge
bowtie2                   2.3.4.1          py35pl5.22.0_0    bioconda
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
cachetools                2.0.1                      py_0    conda-forge
certifi                   2018.4.16                py35_0    conda-forge
cffi                      1.11.5                   py35_0    conda-forge
chardet                   3.0.4                    py35_0    conda-forge
configargparse            0.12.0                   py35_0    conda-forge
cryptography              2.2.1                    py35_0    conda-forge
curl                      7.59.0                        1    conda-forge
datrie                    0.7.1                    py35_0
docutils                  0.14                     py35_0    conda-forge
dropbox                   7.3.1                    py35_0    conda-forge
filechunkio               1.6                      py35_0    bioconda
ftputil                   3.2                      py35_0    bioconda
gmp                       6.1.2                         0    conda-forge
gnutls                    3.5.17                        0    conda-forge
google-auth               1.2.1                      py_0    conda-forge
google-auth-httplib2      0.0.2                    py35_0    conda-forge
google-cloud-core         0.24.1                   py35_0    conda-forge
google-cloud-storage      1.1.1                    py35_0    conda-forge
google-resumable-media    0.0.2                    py35_0    conda-forge
googleapis-common-protos  1.5.3                    py35_0    conda-forge
httplib2                  0.11.1                   py35_0    conda-forge
icu                       58.2                          0    conda-forge
idna                      2.6                      py35_1    conda-forge
idna_ssl                  1.0.0                         0    conda-forge
intel-openmp              2018.0.0                      8
jmespath                  0.9.3                    py35_0    conda-forge
krb5                      1.14.6                        0    conda-forge
libdeflate                0.8                           0    bioconda
libffi                    3.2.1                         3    conda-forge
libgcc                    7.2.0                h69d50b8_2
libgcc-ng                 7.2.0                hdf63c60_3
libgfortran-ng            7.2.0                hdf63c60_3
libidn11                  1.33                          0    conda-forge
libprotobuf               3.5.2                         0    conda-forge
libssh2                   1.8.0                         2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3
megahit                   1.1.2                    py35_1    bioconda
meryl                     2013                          0    bioconda
mkl                       2018.0.2                      1
mkl_fft                   1.0.2                    py35_0    conda-forge
mkl_random                1.0.1                    py35_0    conda-forge
multidict                 4.3.1                    py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
nettle                    3.3                           0    conda-forge
numpy                     1.14.3           py35h14a74c5_0
numpy-base                1.14.3           py35hdbf6ddf_0
openjdk                   8.0.121                       1
openssl                   1.0.2o                        0    conda-forge
pandas                    0.22.0                   py35_1    conda-forge
paramiko                  2.4.1                    py35_0    conda-forge
pcre                      8.41                          1    conda-forge
perl                      5.22.0.1                      0    conda-forge
perl-app-cpanminus        1.7043               pl5.22.0_0    bioconda
perl-archive-tar          2.18                 pl5.22.0_2    bioconda
perl-carp                 1.38                 pl5.22.0_0    bioconda
perl-compress-raw-bzip2   2.069                         1    bioconda
perl-compress-raw-zlib    2.069                         3    bioconda
perl-data-dumper          2.161                pl5.22.0_0    bioconda
perl-exporter             5.72                 pl5.22.0_0    bioconda
perl-exporter-tiny        0.042                         1    bioconda
perl-extutils-makemaker   7.24                 pl5.22.0_1    bioconda
perl-io-compress          2.069                pl5.22.0_2    bioconda
perl-io-zlib              1.10                          1    bioconda
perl-list-moreutils       0.428                pl5.22.0_0    bioconda
perl-pathtools            3.73                          0    bioconda
perl-scalar-list-utils    1.45                          2    bioconda
perl-test-more            1.001002             pl5.22.0_0    bioconda
perl-threaded             5.22.0              pl5.22.0_12    bioconda
pilon                     1.22                     py35_0    bioconda
pip                       9.0.3                    py35_0    conda-forge
prettytable               0.7.2                    py35_1    conda-forge
protobuf                  3.5.2                    py35_0    conda-forge
psutil                    5.4.5                    py35_0    conda-forge
pyasn1                    0.4.2                      py_0    conda-forge
pyasn1-modules            0.0.5                    py35_0    bioconda
pycparser                 2.18                     py35_0    conda-forge
pynacl                    1.1.2                    py35_0    conda-forge
pysftp                    0.2.9                    py35_0    bioconda
python                    3.5.5                         1    conda-forge
python-dateutil           2.3                      py35_0    bioconda
python-irodsclient        0.7.0                      py_0    conda-forge
pytz                      2018.4                     py_0    conda-forge
pyyaml                    3.12                     py35_1    conda-forge
ratelimiter               1.2.0                    py35_0    conda-forge
readline                  7.0                           0    conda-forge
requests                  2.13.0                   py35_0    conda-forge
rsa                       3.1.4                    py35_0    bioconda
s3transfer                0.1.13                   py35_0    conda-forge
samtools                  1.8                           3    bioconda
setuptools                39.1.0                   py35_0    conda-forge
six                       1.11.0                   py35_1    conda-forge
snakemake                 4.8.1                    py35_0    bioconda
sqlite                    3.20.1                        2    conda-forge
tbb                       2018_20171205                 0    conda-forge
tk                        8.6.7                         0    conda-forge
urllib3                   1.12                     py35_0    bioconda
wheel                     0.31.0                   py35_0    conda-forge
wrapt                     1.10.11                  py35_0    conda-forge
xmlrunner                 1.7.7                      py_0    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.7                         0    conda-forge
yarl                      1.2.3                    py35_0    conda-forge
zlib                      1.2.11                        0    conda-forge
nick-youngblut commented 6 years ago

The syntax error in snakemake/metacompass.iter0.py was caused by placing the log: rule after the shell: rule.

Also, there's a syntax error in create_tsv.sh, which creates the following error:

/ebio/abt3_projects/software/dev/tmp/MetaCompass/bin/create_tsv.sh: 22: [: NA: unexpected operator

if [ $recruited_ids !="NA" ] ; then should be changed to if [ $recruited_ids != "NA" ] ; then

nick-youngblut commented 6 years ago

Here's a template .travis-ci file that you could use for testing your repo:

language: python

python:
  - "3.5"

before_install:
  - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
  - chmod +x miniconda.sh                            
  - ./miniconda.sh -b
  - export PATH=$HOME/miniconda3/bin:$PATH
  - conda update --yes conda
  - conda config --add channels bioconda

install:
  - conda install -c bioconda --yes snakemake bowtie2 blast meryl samtools megahit pilon
  - bash install.sh

script:
  - ./go_metacompass.py -r tutorial/Candidatus_Carsonella_ruddii_HT_Thao2000.fasta -P tutorial/thao2000.1.fq,tutorial/thao2000.2.fq -o example1_output -m 3 -t 4
  - wget ftp://public-ftp.hmpdacc.org/Illumina/posterior_fornix/SRS044742.tar.bz2
  - tar xvjf SRS044742.tar.bz2
  - ./go_metacompass.py -P SRS044742/SRS044742.denovo_duplicates_marked.trimmed.1.fastq,SRS044742/SRS044742.denovo_duplicates_marked.trimmed.2.fastq -U SRS044742/SRS044742.denovo_duplicates_marked.trimmed.singleton.fastq -o example2_output

It should work as is, but I haven't tested it.

I can send a PR for this or the syntax error fixes if you'd like.