Open sttongjai opened 1 year ago
It seems one of the gene clusters anvi'o is trying to get alignments for from muscle, is exploding muscle :/ This is an issue anvi'o can do very little. I think there should be a better error message instead of this cryptic output. But essentially I may suggest you to use famsa
as an aligner and see what happens, or skip alignment completely to get your pangenome (this is not a critical step to get your pangenome working).
Alternatively, you can make your genome storage and the exact command available to me via a Dropbox download link, and I can test it :)
@meren just hinting on possibly low effort-high yield replacement of muscle
with PyFamsa.
PyFamsa looks very promising and I feel like perhaps anvi'o should completely switch to that and use famsa via a native Python interface, and dump muscle and all the command line tricks once and for all.
Hello,
I have encountered a BrokenPipeError [Errno 32] near the end of "anvi-pan-genome". I have been using a docker version of ANVIO (pulled the latest version) on MacPro 2019 8-Core Intel Xeon W 3.5 GHz. I am wondering if you could help fixing this issue. The following is what I got after running the following command line.
Thank you very much,
Siripong
ERROR
anvi-pan-genome -g Staphylococcus_aureus-GENOMES.db --project-name "USANEE_20221212" --output-dir StaphAU__USANEE_20221212 --num-threads 14 --minbit 0.5 --mcl-inflation 10 --use-ncbi-blast
WARNING
If you publish results from this workflow, please do not forget to cite DIAMOND (doi:10.1038/nmeth.3176), unless you use it with --use-ncbi-blast flag, and MCL (http://micans.org/mcl/ and doi:10.1007/978-1-61779-361-5_15)
Functions found ..............................:
Genomes storage ..............................: Initialized (storage hash: hashc8e252cd)
Num genomes in storage .......................: 41 Num genomes will be used .....................: 41 Pan database .................................: A new database,
/Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/USANEE_20221212-PAN.db, has been created. Exclude partial gene calls ...................: False
AA sequences FASTA ...........................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa
Num AA sequences reported ....................: 110,924 Num excluded gene calls ......................: 0 Unique AA sequences FASTA ....................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa.unique
WARNING
You elected to use NCBI's
blastp
for amino acid sequence search. Running blastp will be significantly slower than DIAMOND, but in some cases, slightly more sensitive. We are unsure about whether the slight increase in sensitivity may justify significant increase in run time, but you are the boss.NCBI BLAST MAKEDB
BLAST search db ..............................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa.unique
NCBI BLAST SEARCH
BLAST results ................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/blast-search-results.txt
MCL INPUT
Min percent identity .........................: 0.0
Minbit .......................................: 0.5 Filtered search results ......................: 4,899,964 edges stored
MCL input ....................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/mcl-input.txt
MCL
MCL inflation ................................: 10.0 MCL output ...................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/mcl-clusters.txt Number of MCL clusters .......................: 5,983
CITATION
The workflow you are using will likely use 'muscle' by Edgar, doi:10.1093/nar/gkh340 (http://www.drive5.com/muscle) to align your sequences. If you publish your findings, please do not forget to properly credit this tool.
[13 Dec 22 00:01:36 Aligning amino acid sequences for genes in gene clusters] Processed 1214 of 3287 non-singl (...) ETA: 1m56sTraceback (most recent call last): File "/opt/conda/envs/anvioenv/bin/anvi-pan-genome", line 117, in
pan.process()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 925, in process
gene_clusters_dict, unsuccessful_alignments = self.compute_alignments_for_gene_clusters(gene_clusters_dict)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 794, in compute_alignments_for_gene_clusters
gene_clusters_item = output_queue.get()
File "", line 2, in get
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 819, in _callmethod
kind, result = conn.recv()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 253, in recv
buf = self._recv_bytes()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 410, in _recv_bytes
buf = self._recv(4)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 386, in _recv
raise EOFError
EOFError
Process Process-14:
Process Process-9:
Process Process-6:
Process Process-3:
Process Process-7:
Traceback (most recent call last):
Process Process-15:
Traceback (most recent call last):
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, *self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
BrokenPipeError: [Errno 32] Broken pipe
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target( self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-11:
Traceback (most recent call last):
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, *self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target( self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-8:
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, *self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-4:
Process Process-5:
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target( self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-13:
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, *self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-10:
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target( self._args, self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes
self._send(header + buf)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process Process-2:
Traceback (most recent call last):
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker
output_queue.put(output)
File "", line 2, in put
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 400, in _send_bytes
self._send(header)
File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe