Closed dspeth closed 11 months ago
Hey @dspeth, this is a new feature we're testing and I'll ping @Ge0rges here, who implemented this feature just recently.
When I tried to run it with the following command,
anvi-setup-ncbi-cogs --cog-data-dir TEST
and it run without an issue. Can you please confirm that you run it without specifying a --cog-data-dir
flag (as most people will do)?
anvi-setup-ncbi-cogs --reset
seems to be also working for me :/
It would've helped a lot if you were to rerun the command with --debug
flag and send the entire output, @dspeth.
Thank you for your patience.
@meren weird...
I'm indeed running anvi-setup-ncbi-cogs --reset --debug
without specifying a data dir
I did notice that it seems like the original checksum file gets removed by the last line of cogs.py
, but otherwise don't know why the behaviour would be different for me than for you.
could have thought about running it with debug myself here's the debug output:
[speth@login01 ~]$ anvi-setup-ncbi-cogs --reset --debug
COG version ..................................: COG20
COG data source ..............................: The anvi'o default.
COG base directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG
WARNING
===============================================
This program will remove everything in the COG data directory, then download and
reformat everything from scratch.
DOWNLOADING FILE
===============================================
Source URL ...................................: ftp://ftp.ncbi.nih.gov/pub/COG/COG2020/data/cog-20.cog.csv
Output path ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.cog.csv
Downloaded successfully ......................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.cog.csv
DOWNLOADING FILE
===============================================
Source URL ...................................: ftp://ftp.ncbi.nih.gov/pub/COG/COG2020/data/cog-20.def.tab
Output path ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.def.tab
Downloaded successfully ......................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.def.tab
DOWNLOADING FILE
===============================================
Source URL ...................................: ftp://ftp.ncbi.nih.gov/pub/COG/COG2020/data/fun-20.tab
Output path ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/fun-20.tab
Downloaded successfully ......................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/fun-20.tab
DOWNLOADING FILE
===============================================
Source URL ...................................: ftp://ftp.ncbi.nih.gov//pub/COG/COG2020/data/checksums.md5.txt
Output path ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/checksum.md5.txt
Downloaded successfully ......................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/checksum.md5.txt
DOWNLOADING FILE
===============================================
Source URL ...................................: ftp://ftp.ncbi.nih.gov/pub/COG/COG2020/data/cog-20.fa.gz
Output path ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.fa.gz
Downloaded successfully ......................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/RAW_DATA_FROM_NCBI/cog-20.fa.gz
Diamond log ..................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/DB_DIAMOND/log.txt
DIAMOND MAKEDB
===============================================
[DEBUG] `run_command` is running .............: diamond makedb --in /tmp/tmpuk9my6tb -d /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/DB_DIAMOND/COG -p 1
Diamond search DB ............................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/DB_DIAMOND/COG.dmnd
BLAST log ....................................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/DB_BLAST/log.txt
NCBI BLAST MAKEDB
===============================================
[DEBUG] `run_command` is running .............: makeblastdb -in /tmp/tmpuk9my6tb -dbtype prot -out /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/DB_BLAST/COG/COG.fa
BLAST search db ..............................: /tmp/tmpuk9my6tb
@meren, seems like I can rescue this by just putting an empty file named CHECKSUMS.txt
in the cog dir
Still unclear why this not a problem for you
[speth@login01 test]$ anvi-run-ncbi-cogs -c CO36386bin_9_anvio.db -T 8
COG version ..................................: COG20
COG data source ..............................: The anvi'o default.
COG base directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG
COG data directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20
Searching with ...............................: diamond
Directory to store temporary files ...........: /tmp/tmpcpq1soow
Directory will be removed after the run ......: True
DIAMOND BLASTP
===============================================
Additional params for blastp .................:
Search results ...............................: /tmp/tmpcpq1soow/diamond-search-results.txt
DIAMOND VIEW
===============================================
Diamond tabular output file .................: /tmp/tmpcpq1soow/diamond-search-results.txt
COG version ..................................: COG20
COG data source ..............................: The anvi'o default.
COG base directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG
WARNING
===============================================
Some functional annotation sources you wish to add to the database
(COG20_PATHWAY, COG20_FUNCTION, COG20_CATEGORY) are already in the database.
Anvi'o will first drop them so the incoming annotations could REPLACE them.
WARNING
===============================================
Dropping 3 functional annotation sources yo: COG20_PATHWAY, COG20_FUNCTION,
COG20_CATEGORY.
Gene functions ...............................: 7,883 function calls from 3 sources (COG20_PATHWAY, COG20_FUNCTION, COG20_CATEGORY) for 3,437 unique gene calls have been added to the contigs database.
✓ anvi-run-ncbi-cogs took 0:00:15.308512
[speth@login01 test]$ cd /lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/
[speth@login01 COG20]$ ls
total 38M
-rw-r--r-- 1 speth login 949 Aug 29 13:18 CATEGORIES.txt
-rw-r--r-- 1 speth login 0 Aug 29 13:31 CHECKSUMS.txt
-rw-r--r-- 1 speth login 391K Aug 29 13:18 COG.txt
drwxr-xr-x 3 speth login 4 Aug 29 13:19 DB_BLAST
drwxr-xr-x 2 speth login 4 Aug 29 13:18 DB_DIAMOND
-rw-r--r-- 1 speth login 72 Aug 29 13:20 MISSING_COG_IDs.cPickle
-rw-r--r-- 1 speth login 38M Aug 29 13:18 PID-TO-CID.cPickle
drwxr-xr-x 2 speth login 6 Aug 29 13:31 RAW_DATA_FROM_NCBI
[speth@login01 COG20]$ rm CHECKSUMS.txt
[speth@login01 COG20]$ cd
[speth@login01 ~]$ cd test/
[speth@login01 test]$ anvi-run-ncbi-cogs -c CO36386bin_9_anvio.db -T 8
COG version ..................................: COG20
COG data source ..............................: The anvi'o default.
COG base directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG
Config Error: At least one essential formatted file that is necesary for COG operations is not
where it should be
('/lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/CHECKSUMS.txt'). You
should run COG setup, with the flag `--reset` if necessary, to make sure things
are in order.
[speth@login01 test]$ anvi-run-ncbi-cogs -c CO36386bin_9_anvio.db -T 8 --debug
COG version ..................................: COG20
COG data source ..............................: The anvi'o default.
COG base directory ...........................: /lisc/user/speth/github/anvio/anvio/data/misc/COG
Traceback for debugging
#================================================================================
File "/lisc/user/speth/github/anvio/bin/anvi-run-ncbi-cogs", line 65, in <module>
main(args)
File "/lisc/user/speth/github/anvio/anvio/terminal.py", line 881, in wrapper
program_method(*args, **kwargs)
File "/lisc/user/speth/github/anvio/bin/anvi-run-ncbi-cogs", line 35, in main
cogs = COGs(args)
File "/lisc/user/speth/github/anvio/anvio/cogs.py", line 78, in __init__
self.initialize(args)
File "/lisc/user/speth/github/anvio/anvio/cogs.py", line 91, in initialize
self.essential_files = self.COG_setup.get_essential_file_paths()
File "/lisc/user/speth/github/anvio/anvio/cogs.py", line 579, in get_essential_file_paths
"are in order." % essential_files[file_name])
#================================================================================
Config Error: At least one essential formatted file that is necesary for COG operations is not
where it should be
('/lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/CHECKSUMS.txt'). You
should run COG setup, with the flag `--reset` if necessary, to make sure things
are in order.
I'm so sorry, @dspeth. I just realized why anvi-run-ncbi-cogs
is not working :/ While CHECKSUMS.txt
is marked as an essential file, it was removed after the setup and I missed that :/ https://github.com/merenlab/anvio/commit/73b4d98eded3d54f1082c812fa5eb0167c6fadd9 fixes it, but it will require you to re-run the command,
anvi-setup-ncbi-cogs --reset
My apologies. I hope this will solve the issue.
I have a better solution that I'm testing at the moment, but here is the longer story of what is happening, and an explanation for why https://github.com/merenlab/anvio/commit/73b4d98eded3d54f1082c812fa5eb0167c6fadd9 is not the best way to solve this:
The current setup marks the CHECKSUM.txt
as an essential
file:
(...)
'checksum.md5.txt': { # No func as it is called by the setup_raw_data function
'url': 'ftp://ftp.ncbi.nih.gov//pub/COG/COG2020/data/checksums.md5.txt',
'type': 'essential',
'formatted_file_name': 'CHECKSUMS.txt'},
(...)
Which means, at anytime when someone runs anvi-run-ncbi-cogs
, the COGs
class will get an instance of COGsSetup
class which will try to make sure all essential files are in place .. and then it will complain that CHECKSUM.txt
is not there, and will ask people to re-run anvi-setup-ncbi-cogs
(even though it is not necessary for them to do so). The funny thing is, even if they did run anvi-setup-ncbi-cogs
, the code was going to remove CHECKSUMS.txt
, and they were going to be missing this essential file anyway.
The https://github.com/merenlab/anvio/commit/73b4d98eded3d54f1082c812fa5eb0167c6fadd9 solved that latter issue, but I'm now realizing that the best solution is to actually mark CHECKSUMS.txt
as a non-essential
file. This way anyone who already has a working COG setup (like Daan) will continue to use it seamlessly, and the checksum change will only affect people who are just starting to use anvio-dev. I've made that change now with https://github.com/merenlab/anvio/commit/dc0de1ef34e9a5c3243f9c41258d98d2b30c802a, and I think we should be fine :)
We apologize for all the anvio-dev users whose Snakemake workflows and slurm jobs that failed everywhere, and we thank everyone for sticking with anvio-dev anyway.
Best wishes,
@meren thank you for handling this, I apologize for not recognizing the difference between the essential and non-essential parameter there. I did not completely realize the function of that keyword. @dspeth sorry for breaking your installation momentarily!
I think this is a solved issue so I will close it :)
Short description of the problem
anvi-run-ncbi-cogs fails to run because checksum file is missing
anvi'o version
anvio-dev installed via instructions and updated with every time the env is updated
System info
unix life sciences cluster of the university of vienna
Detailed description of the issue
When running anvi-run-ncbi-cogs, I encounted this error:
Config Error: At least one essential formatted file that is necesary for COG operations is not where it should be
('/lisc/user/speth/github/anvio/anvio/data/misc/COG/COG20/CHECKSUMS.txt'). You
should run COG setup, with the flag
--reset
if necessary, to make sure things are in order.After running anvi-setup-ncbi-cogs with the --reset flag, neither the original file "checksum.md5.txt" nor the formatted file "CHECKSUMS.txt" seem to be present.