problem with anvi-pan-genome and diamond

[JDKling]

Hi there!

I'm running your pangenome workflow on some test data and I keep hitting an error when diamond is creating the search database. Here's the result of a self test:

(anvio-5.1) $ anvi-self-test --version
Anvi'o version ...............................: margaret (v5.1)
Profile DB version ...........................: 29
Contigs DB version ...........................: 12
Pan DB version ...............................: 12
Genome data storage version ..................: 6
Auxiliary data storage version ...............: 2
Structure DB version .........................: 1

I'm also working on an HPC running CentOS Linux release 7.5.1804 (Core), and anvio was installed using pip.

Basically everytime I...

anvi-pan-genome -g test-GENOMES.db \
                --project-name "test_VP" \
                --output-dir test_pangenome \
                --num-threads 8 \
                --debug 

It ends up with....

:05:57 DIAMOND] creating the search database (using 8 thread( (...)ESC[0m
Traceback for debugging

  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/bin/anvi-pan-genome", line 109, in <module>
    pan.process()
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/panops.py", line 858, in process
    blastall_results = self.run_search(unique_aas_FASTA_path, unique_aas_names_dict)
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/panops.py", line 259, in run_search
    return self.run_diamond(unique_AA_sequences_fasta_path, unique_AA_sequences_names_dict)
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/panops.py", line 238, in run_diamond
    return diamond.get_blast_results()
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/drivers/diamond.py", line 72, in get_blast_results
    self.makedb()
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/drivers/diamond.py", line 106, in makedb
    utils.run_command(cmd_line, self.run.log_file_path)
  File "/auto/rcf-proj3/jdk/virtual-envs/anvio-5.1/lib/python3.6/site-packages/anvio/utils.py", line 352, in run_command
    raise ConfigError("command was terminated")

Config Error: command was terminated

So far I've tried running anvi-setup-ncbi-cogs --reset, but the problem persists :(

Thanks in advance!

[meren]

HI @JoshuaKling13, this doesn't seem to have anything to do with anvi'o :/ Probably you should get in touch with your sys admin to see why the command is being terminated. Basically something is killing your diamond process. It could be the memory usage limitations set on the HPC you're on.

Best,

[JDKling]

Thanks Meren,

I appreciate the fast response and I’ll get in touch with the folks over at our HPC!

Best, Joshua

On Tue, Aug 28, 2018 at 11:03 AM A. Murat Eren notifications@github.com wrote:

HI @JoshuaKling13 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JoshuaKling13&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=PRwda6R32BsdvCpkGSdoOQ&m=ZdVmTfAKrTak_F_fXFreOwjoC_wbgtmsynkOw8sslcc&s=T3JUxlKHf104ENLJzzJxzKWXJUIHDMwXVWJEBEKExWk&e=, this doesn't seem to have anything to do with anvi'o :/ Probably you should get in touch with your sys admin to see why the command is being terminated. Basically something is killing your diamond process. It could be the memory usage limitations set on the HPC you're on.

Best,

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_merenlab_anvio_issues_947-23issuecomment-2D416684423&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=PRwda6R32BsdvCpkGSdoOQ&m=ZdVmTfAKrTak_F_fXFreOwjoC_wbgtmsynkOw8sslcc&s=cXddt9FXYZlXsIc8K9tCLFG_zos5fbgQ4Mb0ZB-Pr2M&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AnH-5FYvmUj37rexF-2DwObpCDciFHyLdRRtks5uVYYIgaJpZM4WQByc&d=DwMFaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=PRwda6R32BsdvCpkGSdoOQ&m=ZdVmTfAKrTak_F_fXFreOwjoC_wbgtmsynkOw8sslcc&s=bp5DdUfTsqI-WW-686AnF1aTGrnv9yK-NwyXEao5Eyg&e= .

[johanneswerner]

Hi, I just want to mention, I have a similar issue for the metagenome workflow.

anvi-run-ncbi-cogs -c contigs.db -T 28 --sensitive
COG data directory ...........................: /data/miniconda3/envs/anvio/lib/python3.6/site-packages/anvio/data/misc/COG
COG data source ..............................: The anvi'o default.
COG data directory ...........................: /data/miniconda3/envs/anvio/lib/python3.6/site-packages/anvio/data/misc/COG
Searching with ...............................: diamond
Directory to store temporary files ...........: /tmp/tmpvwvqzwik
Directory will be removed after the run ......: True

WARNING
===============================================
182 entries in the sequences table had blank sequences :/ This is related to the
issue at https://github.com/merenlab/anvio/issues/565. If this is like mid-2018
and you still get this warning, please find an anvi'o developer and make them
feel embarrassed. If it is earlier than take this as a simple warning that some
gene calls in your downstream analyses may have no sequences, and that's OK.
This is a very minor issue due to on-the-fly addition of Ribosomal RNA gene
calls to the contigs database, and will likely will not affect anything major.
This warning will go away when anvi'o can seamlessly work with multiple gene
callers (which we are looking forward to implement in the future).

Sequences ....................................: 1101870 sequences reported.
FASTA ........................................: /tmp/tmpvwvqzwik/aa_sequences.fa
DIAMOND is set to be .........................: Sensitive
[30 Aug 18 13:24:27 DIAMOND] running blastp (using 28 thread(s)) ...                                                                                                                          

Config Error: command was terminated

I think, in my case this issue is related to something else (I am not saying it is related to anvi'o), but I am running anvi'o on a virtual server with 28 cores and 230 GB memory (and I am administrating it myself).

Interestingly, the command does not get terminated and runs smoothly if I remove the --sensitive flag.

Any thoughts on that? Thank you so much for your help.

[johanneswerner]

Okay sorry, my /tmp partition was full, same problem as #906.

[meren]

Hi @JoshuaKling13 :) I am sorry I am just catching up with my e-mails. As far as I can see this one resolved itself, but I wanted to say hi and apologize for the late response anyway :)

[JDKling]

Hi @meren!

No worries! You were right that my problem was coming from a memory limitation. Luckily, my sysadmin was willing to allocate more computing resources to let me use anvio. Everything is working like a charm now!

Thanks for pointing me in the right direction!

[loveylbeibei]

i have a problem when i run the anvi-self-test --suite pangenomics it ended up with like the follows:

Config Error: PyANI returned with non-zero exit code, there may be some errors. please check the log file for details.
Config Error: According to the exit code ('255'), anvi'o suspects that something may have gone wrong while running your tests :/ We hope that the reason is clear to you from
the lines above. But if you don't see anything obvious, and especially if the
test ended up running until the end with reasonable looking final results, you
shouldn't worry too much about this error. Life is short and we all can worry
just a bit less.