[E::main] Try to download again the motus profiler

[max1c]

mOTUs 2.5.1 installed via conda.

$ motus profile -f 101L_pe.fq.1.gz -r 101L_pe.fq.2.gz -s 101L_se.fq.gz -o 101L.motus -n 101L -t 32 -c -k species [E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions [E::main] Try to download again the motus profiler

I was also trying to run the tests from: https://motu-tool.org/tutorial.html All of the tests using "test1_single.fastq" run fine. But I cannot find "sample_R1.fq.gz" or "test2_for.fastq" files to run all of the tests.

Thank you.

[AlessioMilanese]

Hi @max1c,

Thanks for your interest in mOTUs.

I was also trying to run the tests from: https://motu-tool.org/tutorial.html All of the tests using "test1_single.fastq" run fine. But I cannot find "sample_R1.fq.gz" or "test2_for.fastq" files to run all of the tests.

That sounds weird. The error that you get is related to the fact that the you cannot load part of the database. If the database has not been downloaded properly by bioconda, you should get this error always, and not just some times.

Can I ask you to run:

motus profile --test

And copy here what you get?

[max1c]

Thanks for the reply @AlessioMilanese Here is the result:

$ motus profile --test
 ------------------------------------------------------------------------------
|                               TEST MOTUS TOOL                                |
 ------------------------------------------------------------------------------

1-- ran setup.py: done

2-- Tools and versions:
- python:    correct
- bwa:       correct
- samtools:  correct
- metaSNV:   correct

3-- Taxonomy profiling test:
- Run motus (-v 1, only error messages):
- end motus call

Check resulting profile: correct

[AlessioMilanese]

Ok, sorry. I need you to run another test!

Can you copy the attached file (test.txt) and then move it to where you want to execute it and in the terminal type:

mv test.txt test.sh
chmod +x test.sh
./test.sh 

And copy what you get here.

[max1c]

which motus? --------------------------------------
/apps/motus/2.5.1/bin/motus
Test --------------------------------------
 ------------------------------------------------------------------------------
|                               TEST MOTUS TOOL                                |
 ------------------------------------------------------------------------------

1-- ran setup.py: done

2-- Tools and versions:
- python:    correct
- bwa:       correct
- samtools:  correct
- metaSNV:   correct

3-- Taxonomy profiling test:
- Run motus (-v 1, only error messages):
- end motus call

Check resulting profile: correct

Profile single --------------------------------------
[main]  MAP_TAX -----------
[main] Number of detected lanes: 1
[main] Run bwa on lane 1
 [map_db](map single reads) 14.41 sec
[main] Total time map_tax: 14.63 sec
[main]  CALC_MGC -----------
[main] Minimum alignment length: 75 (average read length: 100)
 [calc_mgc](parse 1 sam/bam file) 31.27 sec
 [calc_mgc](get mgc abundances) 0.48 sec
[main] Total time calc_mgc: 37.31 sec
[main]  CALC_MOTU -----------
 [calc_motu] (Create taxonomy profile) 0.39 sec
[main]  TOTAL TIME (map_tax+calc_mgc+calc_motu): 52.33 sec
Profile for-rev --------------------------------------
[main]  MAP_TAX -----------
[main] Number of detected lanes: 1
[main] Run bwa on lane 1
 [map_db](map forward reads) 14.78 sec
 [map_db](map reverse reads) 15.42 sec
 [map_db](sort reads) 0.49 sec
[main] Total time map_tax: 30.79 sec
[main]  CALC_MGC -----------
[main] Minimum alignment length: 75 (average read length: 99)
 [calc_mgc](parse 1 sam/bam file) 58.59 sec
 [calc_mgc](get mgc abundances) 0.53 sec
[main] Total time calc_mgc: 69.77 sec
[main]  CALC_MOTU -----------
 [calc_motu] (Create taxonomy profile) 0.37 sec
[main]  TOTAL TIME (map_tax+calc_mgc+calc_motu): 100.93 sec
Profile for-rev-sing ------------------------------------

Program: motus - a tool for marker gene-based OTU (mOTU) profiling
Version: 2.5.1
Reference: Milanese et al. Microbial abundance, activity and population genomic profiling with mOTUs2. Nature Communications (2019). doi: 10.1038/s41467-019-08844-4

Usage: motus <command> [options]

Command:
 -- Taxonomic profiling
      profile     Perform a taxonomic profiling (map_tax + calc_mgc + calc_motu)
      merge       Merge different taxonomic profiles to create a table

      map_tax     Map reads to the marker gene database
      calc_mgc    Aggregate reads from the same marker gene cluster (mgc)
      calc_motu   From a mgc abundance table, produce the mOTU abundance table

 -- SNV calling
      map_snv     Map reads to create bam/sam file for snv calling
      snv_call    SNV calling using metaSNV

Type motus <command> to print the help for a specific command

motus: error: argument -k: invalid choice: 'species' (choose from 'kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'mOTU')
Remove temp data ------------------------------------

[AlessioMilanese]

Ok, I see that you can run mOTUs, and you can run also for and rev.

Your command was:

$ motus profile -f 101L_pe.fq.1.gz -r 101L_pe.fq.2.gz -s 101L_se.fq.gz -o 101L.motus -n 101L -t 32 -c -k species

Can you try to remove the last option, and have:

$ motus profile -f 101L_pe.fq.1.gz -r 101L_pe.fq.2.gz -s 101L_se.fq.gz -o 101L.motus -n 101L -t 32 -c

What happen when you run that?

[max1c]

Still the same:

$ motus profile -f 101L_pe.fq.1.gz -r101L_pe.fq.2.gz -s 101L_se.fq.gz -o 101L.motus -n 101L -t 32 -c [E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions [E::main] Try to download again the motus profiler

[AlessioMilanese]

Ok, maybe it's an issue with the compressed file. Can you try to download the attached file (raw_reads_2.fastq.gz).

and run:

motus profile -s raw_reads_2.fastq.gz -o test.motus

[max1c]

Seems to work. Very strange indeed.

$ motus profile -s raw_reads_2.fastq.gz -o test.motus [main] MAP_TAX ----------- [main] Number of detected lanes: 1 [main] Run bwa on lane 1 [map_db](map single reads) 1.67 sec [main] Total time map_tax: 1.67 sec [main] CALC_MGC ----------- [main] Minimum alignment length: 75 (average read length: 97) [calc_mgc](parse 1 sam/bam file) 2.17 sec [calc_mgc](get mgc abundances) 0.00 sec [main] Total time calc_mgc: 2.28 sec [main] CALC_MOTU ----------- [calc_motu] (Create taxonomy profile) 0.39 sec [main] TOTAL TIME (map_tax+calc_mgc+calc_motu): 4.34 sec

[AlessioMilanese]

I've never seen something like this! Can you run:

motus profile -s 101L_se.fq.gz -o test

So we see if with minimum commands we get the same error.

[max1c]

$ motus profile -s 101L_se.fq.gz -o test [E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions [E::main] Try to download again the motus profiler

$ file 101L_se.fq.gz 101L_se.fq.gz: gzip compressed data, from Unix

$ du -sh 101L_se.fq.gz 779M 101L_se.fq.gz

[AlessioMilanese]

Ok, can I ask if you can try:

zcat 101L_se.fq.gz | head -n 4 > test
gzip test
motus profile -s test.gz 

[max1c]

$ zcat 101L_se.fq.gz | head -n 4 > test $ cat test @CL100108449L2C001R001_440/2 AGTTTAGTGGAGAGAATTTGTTAGCTGAAAAACTTTGGCCATTTTTTTAAGATAAATAAATTGTTGCTACTTTGGTTATTGTTTAGAGCTGAGTCTT + FFFFFFFF=FEFFFEAFFFFEFFFFFFFFFFFFFF?FFF;EFFFFFFBEFFFFFEGFFFFFFFFFFFFFFFFFFFFFGFFFFFG@FEFFFGFFFF@E

$ gzip test $ motus profile -s test.gz [E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions [E::main] Try to download again the motus profiler

Maybe this is a problem with the data?

[AlessioMilanese]

And if you try:

zcat 101L_se.fq.gz | head -n 4 > test
motus profile -s test

[max1c]

Same thing:

$ zcat 101L_se.fq.gz | head -n 4 > test $ motus profile -s test [E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions [E::main] Try to download again the motus profiler

[AlessioMilanese]

That is funny. Can you try to send me the file (test of the last comment)? (at milanese@embl.de)

[max1c]

Sent you an email also attaching here. test.txt

[AlessioMilanese]

It works fine for me. Maybe try to re-install motus.

conda remove motus
conda install -c bioconda motus

[max1c]

$ cd /apps/motus/
$ mv 2.5.1 2.5.1.bak
$ conda create -yp /apps/motus/2.5.1
Collecting package metadata (current_repodata.json): done
Solving environment: done
$ source activate /apps/motus/2.5.1
$ conda install -c bioconda motus
Collecting package metadata (current_repodata.json): done
Solving environment: done
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

$ motus profile -s test
[E::main] Error loading file: /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/db_mOTU_versions
[E::main] Try to download again the motus profiler

$ conda list
 motus              bioconda/noarch::motus-2.5.1-py_0

Not sure what to try here.

[AlessioMilanese]

What you have if you do:

ls /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/

[max1c]

$ ls /apps/motus/2.5.1/share/motus-2.5.1/db_mOTU/ db_mOTU_bam_header_CEN db_mOTU_DB_CEN.fasta.annotations db_mOTU_DB_NR.fasta.amb db_mOTU_genes_length_NR db_mOTU_padding_coordinates_NR.tsv db_mOTU_test db_mOTU_bam_header_NR db_mOTU_DB_CEN.fasta.bwt db_mOTU_DB_NR.fasta.ann db_mOTU_MAP_genes_to_MGCs.tsv db_mOTU_taxonomy_CAMI.tsv db_mOTU_versions db_mOTU_DB_CEN.fasta db_mOTU_DB_CEN.fasta.pac db_mOTU_DB_NR.fasta.bwt db_mOTU_MAP_MGCs_to_mOTUs_in-line.tsv db_mOTU_taxonomy_meta-mOTUs.tsv README db_mOTU_DB_CEN.fasta.amb db_mOTU_DB_CEN.fasta.sa db_mOTU_DB_NR.fasta.pac db_mOTU_MAP_MGCs_to_mOTUs.tsv db_mOTU_taxonomy_ref-mOTUs_short_names.tsv db_mOTU_DB_CEN.fasta.ann db_mOTU_DB_NR.fasta db_mOTU_DB_NR.fasta.sa db_mOTU_padding_coordinates_CEN.tsv db_mOTU_taxonomy_ref-mOTUs.tsv

[AlessioMilanese]

That looks good, there is no issue there.

Let's try to install it with github and see if the issue remains. Move to a dir where you can install motus and type:

git clone https://github.com/motu-tool/mOTUs_v2.git
cd mOTUs_v2
python setup.py
python test.py
cd ..

Then you can run with:

python mOTUs_v2/motus

So that you are sure that you run the github version and not the conda one.

[max1c]

Hi @AlessioMilanese I've finally figure it out. It was an issue with the environment. Once I fixed that everything started working correctly. Thanks for your help.

[AlessioMilanese]

Good, happy that now it works! Happy computation ;)