search

Ecogenomics / CheckM

Assess the quality of microbial genomes recovered from isolates, single cells, and metagenomes
https://ecogenomics.github.io/CheckM/
GNU General Public License v3.0
347 stars 73 forks source link

Fatal exception (source file p7_hmmfile.c, line 2095): hmm write failed. system error: No space left on device #385

Closed samuelmontgomery closed 10 months ago

samuelmontgomery commented 10 months ago

Hi - I keep getting this error when running checkm analyze I have ~700GB of disk space left, about 30GB of RAM, and I am not sure what is going wrong there

`[2024-01-18 03:37:49] INFO: [CheckM - analyze] Identifying marker genes in bins. [2024-01-18 03:37:50] INFO: Identifying marker genes in 9 bins with 16 threads: Fatal exception (source file p7_hmmfile.c, line 2095): Fatal exception (source file p7_hmmfile.c, line 2095): Fatal exception (source file p7_hmmfile.c, line 2095): Fatal exception (source file p7_hmmfile.c, line 2095): Fatal exception (source file p7_hmmfile.c, line 2095): Fatal exception (source file p7_hmmfile.c, line 2095): hmm write failedhmm write failedFatal exception (source file p7_hmmfile.c, line 2095): hmm write failedhmm write failedhmm write failedhmm write failed

hmm write failed

system error: No space left on device

system error: No space left on device system error: No space left on device system error: No space left on device system error: No space left on device system error: No space left on device system error: No space left on device Aborted (core dumped) Aborted (core dumped) Aborted (core dumped) Aborted (core dumped) Aborted (core dumped) Aborted (core dumped) Aborted (core dumped) Fatal exception (source file esl_ssi.c, line 1171): ssi write failed system error: No space left on device

Error: bad file format in HMM file /tmp/8cf1dcd1-2681-41a4-8909-d31d5c3296fe

Error: File format problem in trying to open HMM file /tmp/8cf1dcd1-2681-41a4-8909-d31d5c3296fe. a /tmp/8cf1dcd1-2681-41a4-8909-d31d5c3296fe.ssi file exists (an SSI index), but its SSI format is not recognized

Aborted (core dumped)

Error: File format problem in trying to open HMM file /tmp/cbdc3b45-8e2e-4ef3-82d0-0eb81610c012. a /tmp/cbdc3b45-8e2e-4ef3-82d0-0eb81610c012.ssi file exists (an SSI index), but its SSI format is not recognized

Finished processing 2 of 9 (22.22%) bins.

Error: bad file format in HMM file /tmp/a5230b66-f51e-46ea-bc3b-1fceaf3ee9c8

Error: File format problem in trying to open HMM file /tmp/a5230b66-f51e-46ea-bc3b-1fceaf3ee9c8. a /tmp/a5230b66-f51e-46ea-bc3b-1fceaf3ee9c8.ssi file exists (an SSI index), but its SSI format is not recognized

Error: bad file format in HMM file /tmp/7f0599d3-1968-4bb7-b6a6-d1462489e0e8

Error: File format problem in trying to open HMM file /tmp/7f0599d3-1968-4bb7-b6a6-d1462489e0e8. a /tmp/7f0599d3-1968-4bb7-b6a6-d1462489e0e8.ssi file exists (an SSI index), but its SSI format is not recognized

Error: bad file format in HMM file /tmp/0e5ea67c-0d6c-4e79-b53e-40dcea10c550

Error: File format problem in trying to open HMM file /tmp/0e5ea67c-0d6c-4e79-b53e-40dcea10c550. a /tmp/0e5ea67c-0d6c-4e79-b53e-40dcea10c550.ssi file exists (an SSI index), but its SSI format is not recognized

Error: bad file format in HMM file /tmp/78bb30e2-fbd3-46bc-adcc-d6f52507ed24

Error: File format problem in trying to open HMM file /tmp/78bb30e2-fbd3-46bc-adcc-d6f52507ed24. a /tmp/78bb30e2-fbd3-46bc-adcc-d6f52507ed24.ssi file exists (an SSI index), but its SSI format is not recognized`

Any help is greatly appreciated

samuelmontgomery commented 10 months ago

Have confirmed I am getting this error running checkm test as well

checkm test ~/checkm_test_results [2024-01-18 04:21:16] INFO: CheckM v1.2.2 [2024-01-18 04:21:16] INFO: checkm test /home/ubuntu/checkm_test_results [2024-01-18 04:21:16] INFO: CheckM data: /home/ubuntu/scratch/references/checkm [2024-01-18 04:21:16] INFO: [CheckM - Test] Processing E.coli K12-W3310 to verify operation of CheckM. [2024-01-18 04:21:16] INFO: [Step 1]: Verifying tree command. [2024-01-18 04:21:16] INFO: [CheckM - tree] Placing bins in reference genome tree. [2024-01-18 04:21:16] INFO: Identifying marker genes in 1 bins with 1 threads: Finished processing 1 of 1 (100.00%) bins. [2024-01-18 04:21:33] INFO: Saving HMM info to file. [2024-01-18 04:21:33] INFO: Calculating genome statistics for 1 bins with 1 threads: Finished processing 1 of 1 (100.00%) bins. [2024-01-18 04:21:33] INFO: Extracting marker genes to align. [2024-01-18 04:21:33] INFO: Parsing HMM hits to marker genes: Finished parsing hits for 1 of 1 (100.00%) bins. [2024-01-18 04:21:33] INFO: Extracting 43 HMMs with 1 threads: Finished extracting 43 of 43 (100.00%) HMMs. [2024-01-18 04:21:33] INFO: Aligning 43 marker genes with 1 threads: Finished aligning 43 of 43 (100.00%) marker genes. [2024-01-18 04:21:34] INFO: Reading marker alignment files. [2024-01-18 04:21:34] INFO: Concatenating alignments. [2024-01-18 04:21:34] INFO: Placing 1 bins into the genome tree with pplacer (be patient). [2024-01-18 04:23:08] INFO: { Current stage: 0:01:52.415 || Total: 0:01:52.415 } [2024-01-18 04:23:09] INFO: [Passed] [2024-01-18 04:23:09] INFO: [Step 2]: Verifying tree_qa command. [2024-01-18 04:23:09] INFO: [CheckM - tree_qa] Assessing phylogenetic markers found in each bin. [2024-01-18 04:23:09] INFO: Reading HMM info from file. [2024-01-18 04:23:09] INFO: Parsing HMM hits to marker genes: Finished parsing hits for 1 of 1 (100.00%) bins. [2024-01-18 04:23:10] INFO: QA information written to: /home/ubuntu/checkm_test_results/results/tree_qa_test.tsv [2024-01-18 04:23:10] INFO: { Current stage: 0:00:01.175 || Total: 0:01:53.590 } [2024-01-18 04:23:10] INFO: [Passed] [2024-01-18 04:23:10] INFO: [Step 3]: Verifying lineage_set command. [2024-01-18 04:23:10] INFO: [CheckM - lineage_set] Inferring lineage-specific marker sets. [2024-01-18 04:23:10] INFO: Reading HMM info from file. [2024-01-18 04:23:10] INFO: Parsing HMM hits to marker genes: Finished parsing hits for 1 of 1 (100.00%) bins. [2024-01-18 04:23:10] INFO: Determining marker sets for each genome bin. Finished processing 1 of 1 (100.00%) bins (current: 637000110). [2024-01-18 04:23:10] INFO: Marker set written to: /home/ubuntu/checkm_test_results/results/lineage_set_test.tsv [2024-01-18 04:23:10] INFO: { Current stage: 0:00:00.471 || Total: 0:01:54.062 } [2024-01-18 04:23:10] INFO: [CheckM - lineage_set] Inferring lineage-specific marker sets. [2024-01-18 04:23:10] INFO: Reading HMM info from file. [2024-01-18 04:23:10] INFO: Parsing HMM hits to marker genes: Finished parsing hits for 1 of 1 (100.00%) bins. [2024-01-18 04:23:10] INFO: Determining marker sets for each genome bin. Finished processing 1 of 1 (100.00%) bins (current: 637000110). [2024-01-18 04:23:11] INFO: Marker set written to: /home/ubuntu/checkm_test_results/results/lineage_set_test.tsv [2024-01-18 04:23:11] INFO: { Current stage: 0:00:00.491 || Total: 0:01:54.554 } [2024-01-18 04:23:11] INFO: [Passed] [2024-01-18 04:23:11] INFO: [Step 4]: Verifying analyze command. [2024-01-18 04:23:11] INFO: [CheckM - analyze] Identifying marker genes in bins. [2024-01-18 04:23:11] INFO: Identifying marker genes in 1 bins with 1 threads: Fatal exception (source file p7_hmmfile.c, line 2095): hmm write failed system error: No space left on device Aborted (core dumped)

Error: bad file format in HMM file /tmp/a6510399-9b6d-4624-8d8e-46a4532e1a69

Error: File format problem in trying to open HMM file /tmp/a6510399-9b6d-4624-8d8e-46a4532e1a69. a /tmp/a6510399-9b6d-4624-8d8e-46a4532e1a69.ssi file exists (an SSI index), but its SSI format is not recognized

Finished processing 1 of 1 (100.00%) bins.

[2024-01-18 04:23:31] INFO: Saving HMM info to file. [2024-01-18 04:23:31] INFO: { Current stage: 0:00:20.036 || Total: 0:02:14.590 } [2024-01-18 04:23:31] INFO: Parsing HMM hits to marker genes: [Errno 2] No such file or directory: '/home/ubuntu/checkm_test_results/results/bins/637000110/hmmer.analyze.txt'

[2024-01-18 04:23:31] INFO: Aligning marker genes with multiple hits in a single bin: Finished processing 1 of 1 (100.00%) bins. [2024-01-18 04:23:31] INFO: { Current stage: 0:00:00.042 || Total: 0:02:14.632 } [2024-01-18 04:23:31] INFO: Calculating genome statistics for 1 bins with 1 threads: Finished processing 1 of 1 (100.00%) bins. [2024-01-18 04:23:31] INFO: { Current stage: 0:00:00.163 || Total: 0:02:14.796 } [2024-01-18 04:23:31] INFO: [Passed] [2024-01-18 04:23:31] INFO: [Step 5]: Verifying qa command. [2024-01-18 04:23:31] INFO: [CheckM - qa] Tabulating genome statistics.

[2024-01-18 04:23:31] INFO: Calculating AAI between multi-copy marker genes. [2024-01-18 04:23:31] INFO: Reading HMM info from file. [2024-01-18 04:23:31] INFO: Parsing HMM hits to marker genes: [Errno 2] No such file or directory: '/home/ubuntu/checkm_test_results/results/bins/637000110/hmmer.analyze.txt'

[2024-01-18 04:23:31] INFO: QA information written to: /home/ubuntu/checkm_test_results/results/qa_test.tsv [2024-01-18 04:23:31] INFO: { Current stage: 0:00:00.041 || Total: 0:02:14.838 } [2024-01-18 04:23:31] INFO: [Passed] [2024-01-18 04:23:31] INFO: { Current stage: 0:00:00.000 || Total: 0:02:14.838 }`

samuelmontgomery commented 10 months ago

Some troubleshooting has confirmed that the issue is fixed by setting --tmpdir as there are gigabytes of temp files being written to my instance /tmp/ not on my volume storage, and it is running out of space there