Error running DRAM.py annotate

[bharat1912]

Hi, Thank you for developing DRAM. I have installed the software and the database as follows:

1. Software: installation via conda and DRAM updated. Running version 1.3

2. Database was successfully installed without errors. The output from $DRAM-setup.py print_config is as follows: Processed search databases KEGG db: None KOfam db: /pvol/databases/DRAM_data/kofam_profiles.hmm KOfam KO list: /pvol/databases/DRAM_data/kofam_ko_list.tsv UniRef db: /pvol/databases/DRAM_data/uniref90.20220111.mmsdb Pfam db: /pvol/databases/DRAM_data/pfam.mmspro dbCAN db: /pvol/databases/DRAM_data/dbCAN-HMMdb-V9.txt RefSeq Viral db: /pvol/databases/DRAM_data/refseq_viral.20220111.mmsdb MEROPS peptidase db: /pvol/databases/DRAM_data/peptidases.20220111.mmsdb VOGDB db: /pvol/databases/DRAM_data/vog_latest_hmms.txt

Descriptions of search database entries Pfam hmm dat: /pvol/databases/DRAM_data/Pfam-A.hmm.dat.gz dbCAN family activities: /pvol/databases/DRAM_data/CAZyDB.07302020.fam-activities.txt VOG annotations: /pvol/databases/DRAM_data/vog_annotations_latest.tsv.gz

Description db: /pvol/databases/DRAM_data/description_db.sqlite

DRAM distillation sheets Genome summary form: /pvol/databases/DRAM_data/genome_summary_form.20220111.tsv Module step form: /pvol/databases/DRAM_data/module_step_form.20220111.tsv ETC module database: /pvol/databases/DRAM_data/etc_mdoule_database.20220111.tsv Function heatmap form: /pvol/databases/DRAM_data/function_heatmap_form.20220111.tsv AMG database: /pvol/databases/DRAM_data/amg_database.20220111.tsv

3. The following error when running MAG fasta generated usin metaWRAP - $DRAM.py annotate -i '/pvol/MAG_Analysis/Archaea/*.fa' -o /pvol/MAG_Analysis/Archaea/annotation 3 fastas found 2022-01-11 09:36:27.169674: Annotation started 0:00:00.014588: Retrieved database locations and descriptions 0:00:00.014660: Annotating bin.28.orig 0:00:40.969825: Turning genes from prodigal to mmseqs2 db 0:00:44.021191: Getting hits from kofam 0:17:21.201151: Getting forward best hits from peptidase 0:17:28.535139: Getting reverse best hits from peptidase 0:17:30.016008: Getting descriptions of hits from peptidase /pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/database_handler.py:81: UserWarning: No descriptions were found for your id's. Does this MER0511016 look like an id from peptidase_description warn("No descriptions were found for your id's. Does this %s look like an id from %s" % (list(ids)[0], Traceback (most recent call last): File "/pvol/opt/anaconda3/envs/DRAM/bin/DRAM.py", line 189, in args.func(**args_dict) File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1034, in annotate_bins_cmd annotate_bins(list(set(fasta_locs)), output_dir, min_contig_size, prodigal_mode, trans_table, bit_score_threshold, File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1073, in annotate_bins all_annotations = annotate_fastas(fasta_locs, output_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1007, in annotate_fastas annotate_fasta(fasta_loc, fasta_name, fasta_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 915, in annotate_fasta annotations = annotate_orfs(gene_faa, db_handler, tmp_dir, start_time, custom_db_locs, custom_hmm_locs, File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 816, in annotate_orfs annotation_list.append(do_blast_style_search(query_db, db_handler.db_locs['peptidase'], tmp_dir, File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 679, in do_blast_style_search hits = formater(hits, header_dict) File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 187, in get_peptidase_description header = header_dict[peptidase_hit] KeyError: 'MER0511016'

4. I have also tried the following $DRAM-setup.py update_description_db and got the following output with errors.

Traceback (most recent call last): File "/pvol/opt/anaconda3/envs/DRAM/bin/DRAM-setup.py", line 157, in args.func(**args_dict) File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/database_handler.py", line 331, in populate_description_db db_handler.populate_description_db(output_loc) File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/database_handler.py", line 222, in populate_description_db self.add_descriptions_to_database(self.make_header_dict_from_mmseqs_db(self.db_locs['uniref']), 'uniref', File "/pvol/opt/anaconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/database_handler.py", line 63, in add_descriptions_to_database description_class = TABLE_NAME_TO_CLASS_DICT[db_name] KeyError: 'uniref'

Grateful for any assistance

Regards

[JSSaini]

Same error.

[rmFlynn]

Thanks for all the information! I am looking into this now. And I will update you both soon.

[XC-Zhai]

same error

[zxyEmily]

Hi, Thank you for developing DRAM.I have the same problem, what shouId I do?Do I need to reinstall? 2022-01-20 17:03:35.430900: Annotation started 0:00:00.092935: Retrieved database locations and descriptions 0:00:00.093066: Annotating chr 0:04:42.443821: Turning genes from prodigal to mmseqs2 db 0:05:11.566816: Getting hits from kofam 0:46:13.917040: Getting forward best hits from peptidase 0:46:32.992543: Getting reverse best hits from peptidase 0:46:44.751885: Getting descriptions of hits from peptidase /home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/database_handler.py:81: UserWarning: No descriptions were found for your id's. Does this MER0035513 look like an id from peptidase_description warn("No descriptions were found for your id's. Does this %s look like an id from %s" % (list(ids)[0], Traceback (most recent call last): File "/home/zhouxingya/miniconda3/envs/DRAM/bin/DRAM.py", line 189, in args.func(**args_dict) File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1034, in annotate_bins_cmd annotate_bins(list(set(fasta_locs)), output_dir, min_contig_size, prodigal_mode, trans_table, bit_score_threshold, File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1073, in annotate_bins all_annotations = annotate_fastas(fasta_locs, output_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1007, in annotate_fastas annotate_fasta(fasta_loc, fasta_name, fasta_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 915, in annotate_fasta annotations = annotate_orfs(gene_faa, db_handler, tmp_dir, start_time, custom_db_locs, custom_hmm_locs, File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 816, in annotate_orfs annotation_list.append(do_blast_style_search(query_db, db_handler.db_locs['peptidase'], tmp_dir, File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 679, in do_blast_style_search hits = formater(hits, header_dict) File "/home/zhouxingya/miniconda3/envs/DRAM/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 187, in get_peptidase_description header = header_dict[peptidase_hit] KeyError: 'MER0035513'

Grateful for any assistance

Regards

[genesandatshirt]

Same error here. Thanks for your help!

/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/database_handler.py:81: UserWarning: No descriptions were found for your id's. Does this YP_010051000.1 look like an id from viral_description warn("No descriptions were found for your id's. Does this %s look like an id from %s" % (list(ids)[0], 2022-01-20 10:28:53.718867: Viral annotation started 0:00:00.298671: Retrieved database locations and descriptions 0:00:00.298707: Annotating final-viral-combined-for-dramv 0:00:02.021494: Turning genes from prodigal to mmseqs2 db 0:00:34.333781: Getting hits from kofam 0:02:31.498334: Getting forward best hits from viral 0:02:50.006422: Getting reverse best hits from viral 0:03:03.758527: Getting descriptions of hits from viral Traceback (most recent call last): File "/home/markowrh/.conda/envs/DRAMv/bin/DRAM-v.py", line 153, in args.func(**args_dict) File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_vgfs.py", line 473, in annotate_vgfs annotations = annotate_fastas(contig_locs, output_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1007, in annotate_fastas annotate_fasta(fasta_loc, fasta_name, fasta_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 915, in annotate_fasta annotations = annotate_orfs(gene_faa, db_handler, tmp_dir, start_time, custom_db_locs, custom_hmm_locs, File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 809, in annotate_orfs annotation_list.append(do_blast_style_search(query_db, db_handler.db_locs['viral'], tmp_dir, File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 679, in do_blast_style_search hits = formater(hits, header_dict) File "/home/markowrh/.conda/envs/DRAMv/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 173, in get_basic_description header = header_dict[hit] KeyError: 'YP_010051000.1'

[rmFlynn]

Hi every one, thank you for your incredible patience, testing is a very slow process for this issue. I expect that a new version of DRAM will land in Bioconda any moment now. I hope that it will solve the issues. I will notify you as soon as it hits. Once you upgrade, you may want to run DRAM-setup.py update_description_db. Let me know how things go, if you have any problems, I don't expect the solution to take this long again.

[genesandatshirt]

You rock!!! Will be testing today and reporting back.

[rmFlynn]

Thankyou @genesandatshirt ! Just make sure you are working from the GitHub repo, as I am still waiting on an approval from bioconda to get into that repo. Use DRAM-setup.py version and make sure the output is 1.3.2, you seem to know what you are doing, however.

[rmFlynn]

We are in bioconda! You can now upgrade dram or install with the instructions in the README. You may still want to run DRAM-setup.py version and make sure the output is 1.3.2. Thanks again!

[bharat1912]

Thanks Rory.

[genesandatshirt]

Thanks for the help. I did a reinstall with new version and sadly got the same error with DRAM 1.3.2

/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/database_handler.py:80: UserWarning: No descriptions were found for your id's. Does this YP_010092792.1 look like an id from viral_description warn("No descriptions were found for your id's. Does this %s look like an id from %s" % (list(ids)[0], 2022-01-21 13:06:59.025960: Viral annotation started 0:00:00.016640: Retrieved database locations and descriptions 0:00:00.016674: Annotating final-viral-combined-for-dramv 0:00:01.627227: Turning genes from prodigal to mmseqs2 db 0:00:12.356894: Getting hits from kofam 0:02:31.356036: Getting forward best hits from viral 0:02:59.653995: Getting reverse best hits from viral 0:03:16.090573: Getting descriptions of hits from viral Traceback (most recent call last): File "/home/markowrh/.conda/envs/DRAM_test/bin/DRAM-v.py", line 153, in args.func(**args_dict) File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_vgfs.py", line 473, in annotate_vgfs annotations = annotate_fastas(contig_locs, output_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1007, in annotate_fastas annotate_fasta(fasta_loc, fasta_name, fasta_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 915, in annotate_fasta annotations = annotate_orfs(gene_faa, db_handler, tmp_dir, start_time, custom_db_locs, custom_hmm_locs, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 809, in annotate_orfs annotation_list.append(do_blast_style_search(query_db, db_handler.db_locs['viral'], tmp_dir, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 679, in do_blast_style_search hits = formater(hits, header_dict) File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 173, in get_basic_description header = header_dict[hit] KeyError: 'YP_010092792.1'

[XC-Zhai]

Thanks for the help. I did a reinstall with new version and sadly got the same error with DRAM 1.3.2

/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/database_handler.py:80: UserWarning: No descriptions were found for your id's. Does this YP_010092792.1 look like an id from viral_description warn("No descriptions were found for your id's. Does this %s look like an id from %s" % (list(ids)[0], 2022-01-21 13:06:59.025960: Viral annotation started 0:00:00.016640: Retrieved database locations and descriptions 0:00:00.016674: Annotating final-viral-combined-for-dramv 0:00:01.627227: Turning genes from prodigal to mmseqs2 db 0:00:12.356894: Getting hits from kofam 0:02:31.356036: Getting forward best hits from viral 0:02:59.653995: Getting reverse best hits from viral 0:03:16.090573: Getting descriptions of hits from viral Traceback (most recent call last): File "/home/markowrh/.conda/envs/DRAM_test/bin/DRAM-v.py", line 153, in args.func(**args_dict) File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_vgfs.py", line 473, in annotate_vgfs annotations = annotate_fastas(contig_locs, output_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 1007, in annotate_fastas annotate_fasta(fasta_loc, fasta_name, fasta_dir, db_handler, min_contig_size, prodigal_mode, trans_table, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 915, in annotate_fasta annotations = annotate_orfs(gene_faa, db_handler, tmp_dir, start_time, custom_db_locs, custom_hmm_locs, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 809, in annotate_orfs annotation_list.append(do_blast_style_search(query_db, db_handler.db_locs['viral'], tmp_dir, File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 679, in do_blast_style_search hits = formater(hits, header_dict) File "/home/markowrh/.conda/envs/DRAM_test/lib/python3.9/site-packages/mag_annotator/annotate_bins.py", line 173, in get_basic_description header = header_dict[hit] KeyError: 'YP_010092792.1'

Try to use this for update:DRAM-setup.py update_description_db. It may help with you.

[genesandatshirt]

Thank you. I have this command running. Can you estimate the runtime and RAM for this command?

[XC-Zhai]

Thank you. I have this command running. Can you estimate the runtime and RAM for this command?

if you already have the database, then if just takes you less than 1min, or you need to install all the database, it usaully takes 2 hours based on the instruction from DRAM.

[genesandatshirt]

Interesting. I had to submit it as a cluster job because it used too much RAM on the command line and it's been running for over an hour with no updates to the log.

[genesandatshirt]

*The databases are already installed

[rmFlynn]

It loads much of the database in order to get descriptions, so the memory is large probably similar to the upper bound of the building the db, it should be faster, but it depends more on your disk I/O speed which can be slow for clusters using network storage.

[genesandatshirt]

That explains it. Thanks much! What is the final size of the _db.sqlite file so I can gauge progress?

[rmFlynn]

About 32G