Error: File format problem in trying to open HMM file

Ufungi commented 2 months ago

I installed WITCH through pip on an independent conda environment in Ubuntu 20.04

(witch) genome@limsfep-zen-32c:/data/genome/run/snyoo/programs/appspam/Fungi$ python3 /data/genome/run/snyoo/programs/WITCH/witch.py \

-q Fungi_query.fasta \ -b /data/genome/run/snyoo/programs/WITCH/results/UNITE/Type_UNITE_ref_DB_aligned.fasta \ -e /data/genome/run/snyoo/programs/WITCH/results/UNITE/Type_UNITE_ref_DB_aligned.fasta.raxml.bestTree \ -d /data/genome/run/snyoo/programs/WITCH/results/Fungi \ -o Type_UNITE_ref_DB_query_aligned.fasta \ -t 32 \ --save-weight 1 \ --keep-decomposition 1

***** Configurations ** home.path: /data/genome/run/snyoo/programs/WITCH/witch_msa/home.path main.config: /home/genome/.witch_msa/main.config

    Configs.hmmdir: None
    Configs.input_path: None
    Configs.backbone_path: /data/genome/run/snyoo/programs/WITCH/results/UNITE/Type_UNITE_ref_DB_aligned.fasta
    Configs.backbone_tree_path: /data/genome/run/snyoo/programs/WITCH/results/UNITE/Type_UNITE_ref_DB_aligned.fasta.raxml.bestTree
    Configs.query_path: /data/genome/run/snyoo/programs/appspam/Fungi/Fungi_query.fasta
    Configs.outdir: /data/genome/run/snyoo/programs/WITCH/results/Fungi
    Configs.output_path: /data/genome/run/snyoo/programs/WITCH/results/Fungi/Type_UNITE_ref_DB_query_aligned.fasta
    Configs.chunksize: 1
    Configs.keeptemp: False
    Configs.keep_decomposition: True
    Configs.mode: witch-ng
    Configs.num_hmms: 10
    Configs.use_weight: True
    Configs.save_weight: True
    Configs.alignment_size: 10
    Configs.alignment_upper_bound: None
    Configs.num_cpus: 32
    Configs.max_concurrent_jobs: 320
    Configs.molecule: None
    Configs.magus_path: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/magus.py
    Configs.mafftpath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/mafft/mafft
    Configs.fasttreepath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/fasttree/FastTreeMP
    Configs.mclpath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/mcl/bin/mcl
    Configs.hmmsearchpath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/hmmer/hmmsearch
    Configs.hmmalignpath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/hmmer/hmmalign
    Configs.hmmbuildpath: /data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/hmmer/hmmbuild
    Configs.bypass_setup: False
    Configs.log_path: /data/genome/run/snyoo/programs/WITCH/results/Fungi/log.txt
    Configs.error_path: /data/genome/run/snyoo/programs/WITCH/results/Fungi/error.txt
    Configs.debug_path: /data/genome/run/snyoo/programs/WITCH/results/Fungi/debug.txt
    Configs.runtime_path: /data/genome/run/snyoo/programs/WITCH/results/Fungi/runtime_breakdown.txt
    Configs.keepgcmtemp: False
    Configs.inflation_factor: 4
    Configs.graphclustermethod: mcl
    Configs.graphtracemethod: minclusters
    Configs.graphtraceoptimize: false
    Configs.timeout: 120
    Configs.Backbone: Namespace(alignment_method='magus', alignment_path='', backbone_size='', selection_strategy='median_length', tree_method
    Configs.MAGUS: Namespace(inflationfactor='', graphclustermethod='', graphtracemethod='', graphtraceoptimize='', maxnumsubsets='', mafftpatrograms/WITCH/witch_msa/tools/magus/tools/mafft/mafft', mclpath='/data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/mcl/bin/mcl', hrun/snyoo/programs/WITCH/witch_msa/tools/magus/tools/hmmer/hmmsearch', hmmbuildpath='/data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tignpath='/data/genome/run/snyoo/programs/WITCH/witch_msa/tools/magus/tools/hmmer/hmmalign', fasttreepath='/data/genome/run/snyoo/programs/WITCH/wiasttree/FastTreeMP')

Decomposing the backbone tree... Running...: 72%|██████████████████▊ | 6750/9327 [04:37<01:47, 23.97it/s]Fatal exception (source file ../../src/p7_trace.c, line 165 realloc for size 262144 failed Running...: 73%|██████████████████▉ | 6778/9327 [04:38<01:42, 24.89it/s]Aborted (core dumped) Running...: 100%|██████████████████████████| 9327/9327 [06:30<00:00, 23.86it/s]

Performing all-against-all HMMSearches between the backbone and queries... Running...: 87%|███████████████▌ | 258526/298464 [3:51:46<2:11:32, 5.06it/s]WITCH-NG Error: File format problem in trying to open HMM file /data/genome/run/snyoo/programs/WITCH/results/Fungi/tree_decomp/root/A_00/hmmbuild.model.A File exists, but appears to be empty?

    Running...:  87%|█████████████████▎  | 258529/298464 [3:51:46<35:48, 18.59it/s]

concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/home/genome/anaconda3/envs/witch/lib/python3.12/concurrent/futures/process.py", line 263, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/genome/run/snyoo/programs/WITCH/witch_msa/gcmm/algorithm.py", line 513, in subset_frag_chunk_hmmsearch res = evalHMMSearchOutput(hmmsearch_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/genome/run/snyoo/programs/WITCH/witch_msa/gcmm/algorithm.py", line 558, in evalHMMSearchOutput outfile = open(path, 'r') ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/data/genome/run/snyoo/programs/WITCH/results/Fungi/tree_decomp/root/A_0_0/hmmsearch.resu """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/data/genome/run/snyoo/programs/WITCH/witch.py", line 5, in witch_runner() File "/data/genome/run/snyoo/programs/WITCH/witch_msa/init.py", line 40, in witch_runner mainAlignmentProcess(args) File "/data/genome/run/snyoo/programs/WITCH/witch_msa/gcmm/gcmm.py", line 152, in mainAlignmentProcess hmmsearch_paths = search.search(lock, pool) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/genome/run/snyoo/programs/WITCH/witchmsa/gcmm/algorithm.py", line 298, in search ret, , , = runTasks(func, pool, mytasks, len(subset_args), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/genome/run/snyoo/programs/WITCH/witch_msa/gcmm/task.py", line 126, in runTasks handle_runtime += handleFuture(future, success, ignored, retry, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/genome/run/snyoo/programs/WITCH/witch_msa/gcmm/task.py", line 76, in handleFuture ret = future.result() ^^^^^^^^^^^^^^^ File "/home/genome/anaconda3/envs/witch/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/home/genome/anaconda3/envs/witch/lib/python3.12/concurrent/futures/_base.py", line 401, in get_result raise self._exception FileNotFoundError: [Errno 2] No such file or directory: '/data/genome/run/snyoo/programs/WITCH/results/Fungi/tree_decomp/root/A_0_0/hmmsearch.resu

Error: File format problem in trying to open HMM file /data/genome/run/snyoo/programs/WITCH/results/Fungi/tree_decomp/root/A_00/hmmbuild.model.A File exists, but appears to be empty?

c5shen commented 2 months ago

Hi, WITCH seemed to crash during the initial phase when performing decompositions into subsets and generating HMMs. From the look of the fatal error message:

Fatal exception (source file ../../src/p7_trace.c, line 165 realloc for size 262144 failed

It seems to be a memory issue when running HMMBuild.

This raises a good point for a QOL change to WITCH to sanity check if all HMMs are created correctly in the first phase (and if not, abort the task and notify the user). But to solve your issue, you can try one or more of the following:

Increase the amount of memory available to WITCH
Reduce the amount of cores for less parallelization (-t 16 or -t 24). Heavy parallelization without increasing available memory can easily be an issue.
Try limiting the HMMs to generate. You can control this by setting a lower bound (-A 10 by default) and an upper bound (-Z, by default no upper bound), which says that we are going to decompose to subsets with numbers of sequences $X$, where $A\leq X\leq Z$. If your backbone (Type_UNITE_ref_DB_aligned.fasta.raxml.bestTree in your case) is large, then the largest HMM will be on your entire set of input sequences and could have an impact on memory usage. Intuitively, a small $A$ and large $Z$ gives the best accuracy, but in experiments, the impact usually is negligible for the upper bound (e.g., setting $Z=1000$ should be sufficient).

Ufungi commented 2 months ago

-t 16 & -Z 1000 options worked successfully! Thank you so much 😊

c5shen commented 2 months ago

Glad it worked out!

c5shen / WITCH

Error: File format problem in trying to open HMM file #11