mmseqs2: returned non-zero exit status 1.

[MonicaSteffi]

Dear Developer, I am trying genomad to find taxonomy for my viral contig. Prior to genomad, I performed spades to get assembled contigs and used this as an input for genomad. But I got the following error. This might be associated with to mmseqs2.py

genomad end-to-end --min-score 0.7 --cleanup --splits 8 spade_lim1_1_old/contigs.fasta lim1_1_genomad_output genomad_db

Traceback (most recent call last):
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/genomad/mmseqs2.py", line 131, in run_mmseqs2
    subprocess.run(command, stdout=fout, stderr=fout, check=True)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mmseqs', 'search', PosixPath('lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/query_db/query_db'), PosixPath('/dss/dssfs02/lwp-dss-0001/u7b03/u7b03-dss-0000/ra78zut/genomad_db/genomad_db'), PosixPath('lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/search_db/search_db'), PosixPath('lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/tmp'), '--threads', '56', '-s', '4.2', '--cov-mode', '1', '-c', '0.2', '-e', '0.001', '--split', '8', '--split-mode', '0']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/bin/genomad", line 10, in <module>
    sys.exit(cli())
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/rich_click/rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/genomad/cli.py", line 1208, in end_to_end
    ctx.invoke(
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/genomad/cli.py", line 425, in annotate
    genomad.annotate.main(
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/genomad/modules/annotate.py", line 202, in main
    mmseqs2_obj.run_mmseqs2(threads, sensitivity, evalue, splits)
  File "/dss/dsshome1/0F/ra78zut/miniconda3/envs/genomad/lib/python3.10/site-packages/genomad/mmseqs2.py", line 134, in run_mmseqs2
    raise Exception(f"'{command_str}' failed.") from e
Exception: 'mmseqs search lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/query_db/query_db /dss/dssfs02/lwp-dss-0001/u7b03/u7b03-dss-0000/ra78zut/genomad_db/genomad_db lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/search_db/search_db lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/tmp --threads 56 -s 4.2 --cov-mode 1 -c 0.2 -e 0.001 --split 8 --split-mode 0' failed.

Any help would be appreciated.

[apcamargo]

Hi @MonicaSteffi

There are things that could be causing this:

• You are using a version of MMseqs2 that is incompatible with geNomad. If you installed via conda/mamba that shouldn't be the case. Just to be sure, can you check which versions of geNomad and MMseqs2 you have installed?
• Your machine is running out of memory. In that case, increasing the number of splits should solve the issue (try 12 or 16). Are you running geNomad on a server or a personal computer? Do you know how much memory the machine has available?

Also, geNomad saves a log of MMseqs2 execution at lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/mmseqs2.log. Can you paste it here?

[MonicaSteffi]

Hi @apcamargo ,

Thank you for the reply. MMseqs Version: 14.7e284 genomad version is 1.4.0

• Your machine is running out of memory. In that case, increasing the number of splits should solve the issue (try 12 or 16). Are you running geNomad on a server or a personal computer? Do you know how much memory the machine has available?

I also tried with higher memory, But still the same error

and the mmseq logfile:

`createdb lim1_1_genomad_output/contigs_annotate/contigs_proteins.faa lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/query_db/query_db

MMseqs Version: 14.7e284 Database type 0 Shuffle input database true Createdb mode 0 Write lookup file 1 Offset of numeric ids 0 Compressed 0 Verbosity 3

Converting sequences [ Time for merging to query_db_h: 0h 0m 0s 14ms Time for merging to query_db: 0h 0m 0s 13ms Database type: Aminoacid Time for processing: 0h 0m 0s 52ms search lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/query_db/query_db /dss/dssfs02/lwp-dss-0001/u7b03/u7b03-dss-0000/ra78zut/genomad_db/genomad_db lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/search_db/search_db lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/tmp --threads 56 -s 4.2 --cov-mode 1 -c 0.2 -e 0.001 --split 8 --split-mode 0

MMseqs Version: 14.7e284 Substitution matrix aa:blosum62.out,nucl:nucleotide.out Add backtrace false Alignment mode 2 Alignment mode 0 Allow wrapped scoring false E-value threshold 0.001 Seq. id. threshold 0 Min alignment length 0 Seq. id. mode 0 Alternative alignments 0 Coverage threshold 0.2 Coverage mode 1 Max sequence length 65535 Compositional bias 1 Compositional bias 1 Max reject 2147483647 Max accept 2147483647 Include identical seq. id. false Preload mode 0 Pseudo count a substitution:1.100,context:1.400 Pseudo count b substitution:4.100,context:5.800 Score bias 0 Realign hits false Realign score bias -0.2 Realign max seqs 2147483647 Correlation score weight 0 Gap open cost aa:11,nucl:5 Gap extension cost aa:1,nucl:2 Zdrop 40 Threads 56 Compressed 0 Verbosity 3 Seed substitution matrix aa:VTML80.out,nucl:nucleotide.out Sensitivity 4.2 k-mer length 5 k-score seq:2147483647,prof:2147483647 Alphabet size aa:21,nucl:5 Max results per query 300 Split database 8 Split mode 0 Split memory limit 0 Diagonal scoring true Exact k-mer matching 0 Mask residues 1 Mask residues probability 0.9 Mask lower case residues 0 Minimum diagonal score 15 Selected taxa Spaced k-mers 1 Spaced k-mer pattern Local temporary path Rescore mode 0 Remove hits by seq. id. and coverage false Sort results 0 Mask profile 1 Profile E-value threshold 0.1 Global sequence weighting false Allow deletions false Filter MSA 1 Use filter only at N seqs 0 Maximum seq. id. threshold 0.9 Minimum seq. id. 0.0 Minimum score per column -20 Minimum coverage 0 Select N most diverse seqs 1000 Pseudo count mode 0 Gap pseudo count 10 Min codons in orf 30 Max codons in length 32734 Max orf gaps 2147483647 Contig start mode 2 Contig end mode 2 Orf start mode 1 Forward frames 1,2,3 Reverse frames 1,2,3 Translation table 1 Translate orf 0 Use all table starts false Offset of numeric ids 0 Create lookup 0 Add orf stop false Overlap between sequences 0 Sequence split mode 1 Header split mode 0 Chain overlapping alignments 0 Merge query 1 Search type 0 Search iterations 1 Start sensitivity 4 Search steps 1 Exhaustive search mode false Filter results during exhaustive search 0 Strand selection 1 LCA search mode false Disk space limit 0 MPI runner Force restart with latest tmp false Remove temporary files false

Failed to execute lim1_1_genomad_output/contigs_annotate/contigs_mmseqs2/tmp/8896424563579662339/searchtargetprofile.sh with error 13.`

[apcamargo]

It seems that this is being caused because the filesystem where you are writing the results doesn't allow execution of scripts (see https://github.com/soedinglab/MMseqs2/issues/534). During its execution MMseqs2 generates and runs a couple of scripts, which are failing because of this limitation.

I can try to add an option to geNomad to allow MMseqs2 directory to be written in a separate location. In the meantime, can you try to write the results in a different place (e.g. your home directory)?

[deminatanja]

Hi, I also have the memory issue. I have the error of OSError: [Errno 28] No space left on device, while running geNomad at HPC with over 2 Tb memory on the disk and 300 Gb reservation for the batch job run. I have been using --split 8. What can be tried to solve the memory issue? Btw, I also tried running geNomad at NMDC EDGE and there has also been a memory limit error, although the input file was much smaller, and there, one can't choose much when submitting a job. Best regards, Tatiana

[apcamargo]

Hi @deminatanja

I also have the memory issue. I have the error of OSError: [Errno 28] No space left on device, while running geNomad at HPC with over 2 Tb memory on the disk and 300 Gb reservation for the batch job run. I have been using --split 8. What can be tried to solve the memory issue?

I don't think this a memory issue. No space left on device means you don't have enough storage space. Have you checked your disk usage?

Btw, I also tried running geNomad at NMDC EDGE and there has also been a memory limit error, although the input file was much smaller, and there, one can't choose much when submitting a job.

Can you send me the log?

[deminatanja]

The device storage space is over 2 Tb available, so should be enough...

Here is the log from NMDC EDGE run:

Generate WDL and inputs json submit workflow to cromwell Cromwell job status: Running Cromwell job status: Failed viral.gn Traceback (most recent call last): File "/opt/conda/bin/genomad", line 10, in sys.exit(cli()) File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(*args, kwargs) File "/opt/conda/lib/python3.9/site-packages/rich_click/rich_group.py", line 21, in main rv = super().main(args, standalone_mode=False, kwargs) File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, *kwargs) File "/opt/conda/lib/python3.9/site-packages/genomad/cli.py", line 1023, in end_to_end ctx.invoke( File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, **kwargs) File "/opt/conda/lib/python3.9/site-packages/genomad/cli.py", line 338, in annotate genomad.annotate.main( File "/opt/conda/lib/python3.9/site-packages/genomad/modules/annotate.py", line 178, in main prodigal_obj.run_parallel_prodigal(threads) File "/opt/conda/lib/python3.9/site-packages/genomad/prodigal.py", line 92, in run_parallel_prodigal self._append_prodigal_fasta(current_file_path, protid_start) File "/opt/conda/lib/python3.9/site-packages/genomad/prodigal.py", line 42, in _append_prodigal_fasta match.group(1) AttributeError: 'NoneType' object has no attribute 'group' slurmstepd: error: Detected 17 oom-kill event(s) in StepId=20795329.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

[deminatanja]

Here is also a full log from the HPC run:

[22:14:27] Executing genomad annotate. Traceback (most recent call last): File "/projappl/project_2006548/genomad/bin/genomad", line 10, in sys.exit(cli()) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1130, in call return self.main(*args, kwargs) File "/opt/conda/lib/python3.10/site-packages/rich_click/rich_group.py", line 21, in main rv = super().main(args, standalone_mode=False, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, *kwargs) File "/opt/conda/lib/python3.10/site-packages/genomad/cli.py", line 1208, in end_to_end ctx.invoke( File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/genomad/cli.py", line 425, in annotate genomad.annotate.main( File "/opt/conda/lib/python3.10/site-packages/genomad/modules/annotate.py", line 178, in main prodigal_obj.run_parallel_prodigal(threads) File "/opt/conda/lib/python3.10/site-packages/genomad/prodigal.py", line 78, in run_parallel_prodigal current_file.write(line) OSError: [Errno 28] No space left on device

[apcamargo]

Ok. These issues seem to be distinct.

The error you got in your HPC is most likely not memory. It is failing during the prodigal-gv execution step (which uses very little memory) while writing a file. It does seem that, for some reason, the process is being killed because you are out of storage. How big is the input (in number of sequences and average sequence length)?

There seems to be a problem with memory in NMDC Edge. I'll try to get this solved as quick as possible.

[deminatanja]

Here is some statistics about the input file:

contigs 1275686 contigs (>= 0 bp) 4189226 contigs (>= 1000 bp) 376034 contigs (>= 5000 bp) 19849 contigs (>= 10000 bp) 4814 contigs (>= 25000 bp) 584 contigs (>= 50000 bp) 115 Largest contig 189326 Total length (>= 1000 bp) 802707590 Total length (>= 5000 bp) 184943840 Total length (>= 10000 bp) 84208373 Total length (>= 25000 bp) 24221139 Total length (>= 50000 bp) 8424764 N50 1172 N75 714 L50 284205 L75 678615

[apcamargo]

Ok. The input is pretty big, so maybe you are running out of storage when writing the outputs? What's the output of df -h?

Another option is to just split your input and run geNomad in batches to avoid this sort of problem.

[MonicaSteffi]

Hi @apcamargo Thank you. It worked when I changed the output directory

Regards Monica

[apcamargo]

No problems :)

[MonicaSteffi]

No problems :)

How do i get the gtf or gff3 files for annotation which can be visualized using any software?

[deminatanja]

Ok. The input is pretty big, so maybe you are running out of storage when writing the outputs? What's the output of df -h?

Another option is to just split your input and run geNomad in batches to avoid this sort of problem.

The disc resources are (used/total): 351G/3.0T, 3.8M/10M files.

I was testing geNomad with a smaller input file, but ran into another error. Please see a separate issue opened here.

[apcamargo]

@MonicaSteffi You can use the script below to convert geNomad's tabular gene file to a GFF:

chmod +x convert_tabular_to_gff.py
# ./convert_tabular_to_gff.py [INPUT] [OUTPUT]
./convert_tabular_to_gff.py genomad_output/metagenome_summary/metagenome_plasmid_genes.tsv plasmid.gff

Outputting GFF files is pretty useful. I might make geNomad output GFF files by default in a future update.

convert_tabular_to_gff.py

```python #!/usr/bin/env python3 import sys from collections import namedtuple Row = namedtuple( "Row", [ "gene", "start", "end", "length", "strand", "gc_content", "genetic_code", "rbs_motif", "marker", "evalue", "bitscore", "uscg", "plasmid_hallmark", "virus_hallmark", "taxid", "taxname", "annotation_conjscan", "annotation_amr", "annotation_accessions", "annotation_description", ], ) input = sys.argv[1] output = sys.argv[2] with open(input) as fin, open(output, "w") as fout: next(fin) for row in fin: row = row.strip("\n").split("\t") row = Row(*row) fout.write(f"{row.gene.rsplit('_', 1)[0]}\t") fout.write(f".\tCDS\t{row.start}\t{row.end}\t.\t") fout.write("+\t.\t" if row.strand == "1" else "-\t.\t") fout.write(f"ID={row.gene};Name={row.gene};") fout.write(f"gc_content={row.gc_content};genetic_code={row.genetic_code};") fout.write(f"rbs_motif={row.rbs_motif};marker={row.marker};") fout.write(f"uscg={row.uscg};plasmid_hallmark={row.plasmid_hallmark};") fout.write(f"virus_hallmark={row.virus_hallmark};taxid={row.taxid};") fout.write(f"taxname={row.taxname};annotation_conjscan={row.annotation_conjscan};") fout.write(f"annotation_amr={row.annotation_amr};") fout.write(f"annotation_accessions={row.annotation_accessions.replace(';', ',')};") fout.write(f"annotation_description={row.annotation_description.replace(';', ',')}\n") ```

[asierFernandezP]

Hi,

I am also getting the same error (non-zero exit status 1) since I updated to the last version of geNomad. I tried to change the directory of the output but I still get the same error. Here is the content of the log file:

[10:06:18] Executing genomad annotate.                                          
[10:06:18] Creating the                                                         
           /home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annota
           te directory.                                                        
[15:19:05] Proteins predicted with prodigal-gv were written to                  
           all_predicted_viral_contigs_proteins.faa.                            
Traceback (most recent call last):
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/genomad/mmseqs2.py", line 131, in run_mmseqs2
    subprocess.run(command, stdout=fout, stderr=fout, check=True)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mmseqs', 'createdb', PosixPath('/home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annotate/all_predicted_viral_contigs_proteins.faa'), PosixPath('/home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annotate/all_predicted_viral_contigs_mmseqs2/query_db/query_db')]' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/umcg-afernandez/.conda/envs/genomad/bin/genomad", line 10, in <module>
    sys.exit(cli())
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/rich_click/rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/genomad/cli.py", line 1208, in end_to_end
    ctx.invoke(
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/genomad/cli.py", line 425, in annotate
    genomad.annotate.main(
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/genomad/modules/annotate.py", line 202, in main
    mmseqs2_obj.run_mmseqs2(threads, sensitivity, evalue, splits)
  File "/home/umcg-afernandez/.conda/envs/genomad/lib/python3.10/site-packages/genomad/mmseqs2.py", line 134, in run_mmseqs2
    raise Exception(f"'{command_str}' failed.") from e
Exception: 'mmseqs createdb /home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annotate/all_predicted_viral_contigs_proteins.faa /home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annotate/all_predicted_viral_contigs_mmseqs2/query_db/query_db' failed.

[apcamargo]

Hi @asierFernandezP

Can you share the contents of /home/umcg-afernandez/cons_geNomad/all_predicted_viral_contigs_annotate/all_predicted_viral_contigs_mmseqs2/mmseqs2.log? Did it work in the previous version?

[asierFernandezP]

Hi,

Thanks for the quick answer! I am sorry I deleted this file, but after rerunning it 4 times (without any changes) it worked. This problem only appeared after reinstalling geNomad using conda (24-02-2023) in order to get the --conservative option, not present in the previous version. But, as I said, it worked after a few trials without any modifications. I will let you know if I see this problem again.

[apcamargo]

Good to know that you didn't have any problems again, @asierFernandezP. If you are interested in using the --conservative flag, might be worth to take a quick look here: https://portal.nersc.gov/genomad/post_classification_filtering.html#default-parameters-and-presets

I'll close this issue for now.