WARNING: At least one BLAST run failed. ANIb may fail and The condensed distance matrix must contain only finite values

genomesandMGEs commented 3 years ago

Hey, I got an error while running ANIb on a large dataset (>2k bacterial genomes). I installed pyani via conda, version 0.2.10 and ran this command:

average_nucleotide_identity.py -i ./ -o ANIb_output -g --gformat svg,png -m ANIb --gmethod seaborn

This is the error I got:

WARNING: At least one BLAST run failed. ANIb may fail. /gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py:649: UserWarning: Clustering large matrix with scipy. Installing fastcluster may give better performance. warnings.warn(msg) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/bin/average_nucleotide_identity.py", line 977, in draw(methods[args.method][1], gfmt) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/bin/average_nucleotide_identity.py", line 809, in draw pyani_graphics.heatmap_seaborn( File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/pyani/pyani_graphics.py", line 200, in heatmap_seaborn fig = get_seaborn_clustermap(dfr, params, title=title) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/pyani/pyani_graphics.py", line 144, in get_seaborn_clustermap fig = sns.clustermap( File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/_decorators.py", line 46, in inner_f return f(kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 1408, in clustermap return plotter.plot(metric=metric, method=method, File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 1221, in plot self.plot_dendrograms(row_cluster, col_cluster, metric, method, File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 1066, in plot_dendrograms self.dendrogram_row = dendrogram( File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/_decorators.py", line 46, in inner_f return f(kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 774, in dendrogram plotter = _DendrogramPlotter(data, linkage=linkage, axis=axis, File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 584, in init self.linkage = self.calculated_linkage File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 651, in calculated_linkage return self._calculate_linkage_scipy() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/seaborn/matrix.py", line 619, in _calculate_linkage_scipy linkage = hierarchy.linkage(self.array, method=self.method, File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/site-packages/scipy/cluster/hierarchy.py", line 1065, in linkage raise ValueError("The condensed distance matrix must contain only " ValueError: The condensed distance matrix must contain only finite values.

Thanks!

widdowquinn commented 3 years ago

Hi @genomesandMGEs - thank you for your interest in pyani.

I expect the issue is as the warning says: a BLAST run failed. Failure here includes writing no output, which can happen when there is no identifiable homology between two genomes (pyani may not distinguish between a BLAST run halting early, and a BLAST run producing zero matches). It is possible that, with a dataset of 2k bacterial genomes, you have at least one pair of genomes with this issue.

In your position, I would identify the pair(s) of genomes with no identifiable homology and modify the input dataset accordingly. There are several methods for achieving this, including running a tool like sourmash or fastANI to quickly identify very distant pairs of genomes.

I hope this is helpful to you.

L.

genomesandMGEs commented 3 years ago

Hi @widdowquinn ,

Thanks for the quick reply. I find it difficult that some pair of genomes have no identifiable homology, since all these genomes belong to the sames species, and I used a filter to only include genomes with a maximum distance to the reference of 0.05 (~95% ANI).

Is it possible that ANIb can't handle such large dataset? I'm trying to run with ANIm now to see if it works.

widdowquinn commented 3 years ago

I have run pyani on similarly-sized datasets, so I would not expect this to be the issue.

When you determine genome distance, do you take coverage/alignments fraction into account? (It’s possible to get falsely high identities because a very small region of genome is being aligned, for instance)

It’s worth keeping in mind also that existing species assignments can be inaccurate.

If you can identify the specific BLAST output that gives the error, that may help the diagnosis.

genomesandMGEs commented 3 years ago

That's a good point - I used panacota to donwload the genomes of interest and filter by the 0.05 max distance, but there's no mention of the coverage/alignment fraction cut-off. I just ran bindash on my 2009 genomes, and here's the first 10 hits. Table was sorted by col 5, that represents shared k-mers/total k-mers:

PSAE_0321_00764.fasta PSAE_0321_00952.fasta 2.8550e-02 0.0000e+00 0.378418 PSAE_0321_00150.fasta PSAE_0321_01107.fasta 2.8372e-02 0.0000e+00 0.380371 PSAE_0321_01067.fasta PSAE_0321_01107.fasta 2.8108e-02 0.0000e+00 0.383301 PSAE_0321_01107.fasta PSAE_0321_01613.fasta 2.8108e-02 0.0000e+00 0.383301 PSAE_0321_00952.fasta PSAE_0321_01751.fasta 2.8064e-02 0.0000e+00 0.383789 PSAE_0321_00232.fasta PSAE_0321_01107.fasta 2.7976e-02 0.0000e+00 0.384766 PSAE_0321_00764.fasta PSAE_0321_00926.fasta 2.7933e-02 0.0000e+00 0.385254 PSAE_0321_00956.fasta PSAE_0321_01107.fasta 2.7933e-02 0.0000e+00 0.385254 PSAE_0321_00764.fasta PSAE_0321_01791.fasta 2.7889e-02 0.0000e+00 0.385742 PSAE_0321_01107.fasta PSAE_0321_01651.fasta 2.7846e-02 0.0000e+00 0.38623

So, the genome pairs with the smallest fraction of shared k-mers is ~38%. Do you think this will be problematic for pyani's ANIb?

widdowquinn commented 3 years ago

You'll have a better idea of that if you, for instance, put those genomes into a folder with your reference and run ANIb, then inspect the BLAST output that gets written. If you use pyani's log output you should be able to reproduce each individual BLAST command, too to see if the BLAST command generates usable output outwith the context of pyani.

If BLAST falls over for whatever reason and doesn't produce an output, this will be problematic for pyani's implementation of ANIb; if there's not sufficient similarity between the genomes that is problematic for the ANIb algorithm in general.

widdowquinn commented 3 years ago

FWIW I try not to encourage ANIb - the ANIm algorithm is more robust and stable in part because it doesn't require arbitrary fragmentation of genomes, or for properties of the alignments of those specific (yet arbitrary) fragments to be met.

genomesandMGEs commented 3 years ago

Just a heads up - just ran ANIm using average_nucleotide_identity.py -i ./ -o ANIm_output -g --gformat svg,png --gmethod seaborn and also got an error:

Process ForkPoolWorker-20: Process ForkPoolWorker-24: Process ForkPoolWorker-6: Process ForkPoolWorker-4: Process ForkPoolWorker-10: Process ForkPoolWorker-22: Process ForkPoolWorker-26: Process ForkPoolWorker-3: Process ForkPoolWorker-19: Process ForkPoolWorker-14: Process ForkPoolWorker-28: Process ForkPoolWorker-11: Process ForkPoolWorker-27: Process ForkPoolWorker-9: Process ForkPoolWorker-7: Process ForkPoolWorker-5: Process ForkPoolWorker-30: Process ForkPoolWorker-29: Process ForkPoolWorker-16: Process ForkPoolWorker-8: Process ForkPoolWorker-23: /var/spool/slurmd/job894724/slurm_script: line 17: 2828435 Killed average_nucleotide_identity.py -i ./ -o ANIm_output -g --gformat svg,png --gmethod seaborn Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) Traceback (most recent call last):

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) Traceback (most recent call last):

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() Traceback (most recent call last): Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped)))

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, self._kwargs) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-31: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-18: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-2: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-32: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-25: Process ForkPoolWorker-12: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-21: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-17: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-15: Process ForkPoolWorker-13: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 410, in _send_bytes self._send(buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 409, in _send_bytes self._send(header) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-1: Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size]) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf) File "/gxfs_home/cau/sunzm592/anaconda3/envs/pyani/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

widdowquinn commented 3 years ago

Can you get pyani to run on a smaller dataset with either ANIb or ANIm. It seems odd that you'd have so many broken pipes. Can you check that there is sufficient disk space to write the necessary output files for the analysis (2000 x 2000 x however large the outputs are)?

genomesandMGEs commented 3 years ago

Yes, tried using ANIm just now on a way smaller dataset (n=20), and it worked perfectly. I'm running this command as a batch process in a server, so I guess disk space won't be a problem. Or do you think I should contact the IT support and check if there's some kind of limitation for this?

widdowquinn commented 3 years ago

Looking at your output it appears like you're using SLURM. There may be some issues with how multiprocessing is requesting/using cores which conflicts with that, giving you the broken pipes. There's a PR in on the development version of pyani for SLURM integration, but it's not part of v.0.2. Maybe that's part of the problem, here?

genomesandMGEs commented 3 years ago

Thank you for the tip. I contacted the IT support and they told me that from the Slurm accounting, it seems not enough RAM was available to run this job. So, I extended it to a max of 1400GB and ran ANIm for 5 days as a batch process, but the job didn't finish. You said you ran similar size datasets before using this tool; can you please let me know how long it took to complete the task?

widdowquinn commented 3 years ago

For a set of 1680 genomes:

INFO: Waiting for SGE-submitted jobs to finish (polling)
INFO: ...jobs complete
INFO: Adding comparison results to database...
INFO: Committing results to database
INFO: ...database updated.
INFO: Completed. Time taken: 617381.333

which comes out at around 7d, 3.5h.

genomesandMGEs commented 3 years ago

Thanks for sharing this, will try to run again using a longer duration

genomesandMGEs commented 3 years ago

So, the job failed after 7 days running in the cluster server (on a bigmem node ~1400GB with 32 cpus), with an out of memory error. Do you suggest any of the following:

reduce the memory usage for this analysis, e.g., by chunking the data under consideration into smaller amounts such that less memory may be required;
would the usage of less cpu cores lead to less memory consumption? If yes, one could try to run the analysis on a smaller set of cores though this would increase the overall runtime;

Thanks in advance for your time!

widdowquinn commented 3 years ago

Which stage was pyani at?

If it's falling over during BLAST runs, there's nothing much we can do to make BLAST more efficient.

If it's falling over after the comparison runs are finished, then it may be that the internal data structures holding the compiled outputs are too large to be processed. This compilation step takes place on the node you run pyani from. If you're running pyani from the head/login node of a cluster, which usually has less memory than the worker nodes, this can be an issue. If you can qrsh (or equivalent) into a worker node with more memory and run the tool from there, this might avoid the limited memory overhead on a head node.

The error message you receive would be helpful for diagnosis, if you still have it.

(BTW we're working to avoid that issue for larger datasets in v0.3, by working towards asynchronous update of a local database when each comparison ends - we know the current design has this kind of problem, which is a bottleneck for time as well)

genomesandMGEs commented 3 years ago

Thanks (again) for the quick and detailed reply. Here's the error output:

ANIm_error.txt

Please let me know how I should proceed, and I can then discuss this with the IT support.

widdowquinn commented 3 years ago

That looks like it's falling over during comparison runs.

It looks like a SLURM weirdness to me. I must confess I don't understand it properly, yet - and my SysAdmin friends tell me that local configurations do, apparently, make a difference. I've had similar troubles on SLURM with raxml-ng this week, and I found balancing SLURM settings to be a bit of a chore.

If it helps your IT guys diagnose, then what happens is:

pyani uses multiprocessing to submit as many processes as the local machine can handle (determined by available threads). All downstream memory/thread use is then in the hands of the individual job. If you ask, e.g. BLAST to use 8 threads, then each of the multiprocessing calls will be unaware of this and it's possible to oversubscribe CPUs accidentally.
once all jobs finish, pyani collates the output into an internal representation, and writes its output

I think your runs are failing somewhere in part 1, but I don't know why, I'm afraid.

We're looking at a PR for v0.3 which specifically integrates with SLURM, but it's not on the main branch yet. Fingers crossed that might solve your issue, when it's ready. I'm sorry you're having these troubles right now, though.

baileythegreen commented 3 years ago

I agree it seems like the issue is on the SLURM side of things, probably still to do with the resources being requested versus what is actually needed.

Have you tried this?:

$ sacct --jobs=your_job-id

I am more familiar with SGE schedulers, but this should output a bunch of information about the job, and if it's similar to SGE's qacct, that should include information about resources (time, memory, cores) that were requested, and how much was actually used. I think it'll be more useful than the Python traceback for finding the problem.

If you post it here, maybe we'll spot something useful; but if you talk to IT, you should definitely share the output from that.

widdowquinn commented 3 years ago

Good idea re: sacct, @baileythegreen. You can control what information you get back with sacct, too. I've aliased

sacct -j <jobid> --units=G --format JobID,MaxVMSize,MaxRSS,NodeList,AllocCPUS,TotalCPU,State,Start,End

where MaxVMSize is the amount of memory that is assigned (even if not used).

One of my colleagues had a weird issue recently where they requested 16GB per task, and were somehow allocated 500GB. There was a discussion that followed about how to assign memory with SLURM jobs, and this came from one of the sysadmins:

what slurm means by a cpu depends on how it's configured, in our case cpu=thread, so --mem-per-cpu should be mem per thread, where as --mem should be per task (ie the whole job if it's a simple single job or per each job in an array

I don't know enough about SLURM to advise on good settings here, though.

genomesandMGEs commented 3 years ago

Thank you both for sharing your thoughts. I had a talk with the IT support, and they said this confirms that the calculation is using extreme amount of memory and was killed by slurm due to out of memory. According to them, the real question is why it is using so much memory and if this is reasonable with 1300GB+. They argue the problem is independent of Slurm. They asked me to share the output from the top command and the figure they produced in the pdf file:

output_top.txt show_edited_replicons.pdf

Also, they warned me about one process in the list below

3475335 szrzs212 20 0 510.6g 505.3g 23012 S 0.0 33.5 49:42.26 average_nucleot

that is continuously accumulating main memory. Maybe that is some kind of bug or this process is not correctly handled by pyani?

Thanks again!

widdowquinn commented 3 years ago

There are a couple of things to comment on here, I think.

The first is that currently we don't support parallelisation with SLURM directly in pyani (in part because I've only recently got access to a cluster with a SLURM scheduler).

What really needs to happen is that each individual pairwise comparison should be carried out as a single job in an array of jobs (this is how the SGE scheduler currently works). I would expect that to keep memory requirements local to each comparison. Because of the lack of SLURM support in pyani you're having to treat your node like a single machine and use multiprocessing to manage multiple simultaneous jobs on separate threads (more info here. They are all, however, still controlled by the driver script, which must manage them and it appears that this generates a very large memory requirement for this driver script.

It looks like that task of managing all the necessary comparisons using multiprocessing is too big, even for the machine you're running on. I'm surprised, but if that's the case there's not much to be done with the code you're currently running.

However, we do have a PR (#236) for SLURM support that is not yet tested or integrated into the main codebase. This should enable array jobs using SLURM in the same way we currently do for SGE. I would expect that to solve your problem. Using this would mean moving up to v0.3, but this is not available through conda and you would need to install from this repository.

I appreciate this isn't the positive answer you're looking for to move your analysis on quickly, but I think it's where we are, just now. I want to get SLURM support going in the main code quite soon, so hopefully you won't need to wait too long.

Many apologies,

L.

widdowquinn / pyani

WARNING: At least one BLAST run failed. ANIb may fail and The condensed distance matrix must contain only finite values #267