search

czbiohub-sf / MIDAS

Metagenomic Intra-Species Diversity Analysis (MIDAS)
MIT License
35 stars 10 forks source link

Errors when download the MIDASdb #138

Closed yejunbin closed 2 months ago

yejunbin commented 3 months ago

midas2 database --init --midasdb_name uhgg --midasdb_dir /Disk2/database/midas2/uhgg

1717129948.0: Initate MIDASDB uhgg to /Disk2/database/midas2/uhgg::start 1717130181.8: Initate MIDASDB uhgg to /Disk2/database/midas2/uhgg::finish 1717130181.8: Download chunks for MIDASDB uhgg to /Disk2/database/midas2/uhgg::start

Traceback (most recent call last): File "/Disk1/mambaforge/envs/midasv3/bin/midas2", line 10, in sys.exit(main()) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/main.py", line 25, in main return subcommand_main(subcommand_args) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/subcommands/database.py", line 146, in main init_midasdb(args) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/subcommands/database.py", line 29, in init_midasdb midasdb.fetch_files("chunks") File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/models/midasdb.py", line 190, in fetch_files return self.fetch_tarball(filename, list_of_species) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/models/midasdb.py", line 251, in fetch_tarball _fetched_file = self.get_target_layout(ct, False, sid, gid, DEFAULT_CHUNKS[i]) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/models/midasdb.py", line 165, in get_target_layout local_path = self.construct_local_path(filename, species_id, genome_id, component) File "/Disk1/mambaforge/envs/midasv3/lib/python3.9/site-packages/midas2/models/midasdb.py", line 152, in construct_local_path local_path = _get_local_path(get_midasdb_layout(species_id, genome_id, component)[filename], self.db_dir) KeyError: 'chunks_centroids

midas2 database --download --midasdb_name uhgg --midasdb_dir /Disk2/database/midas2/uhgg --species all

1717128183.4: Downloading MIDAS database for sliced species 7 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 2 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 13 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 9 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 6 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 12 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 11 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 0 with 14 cores in total::start 1717128183.5: Downloading MIDAS database for sliced species 10 with 14 cores in total::start 1717128183.6: Downloading MIDAS database for sliced species 5 with 14 cores in total::start 1717128183.6: Downloading MIDAS database for sliced species 3 with 14 cores in total::start 1717128183.6: Downloading MIDAS database for sliced species 4 with 14 cores in total::start 1717128183.6: Downloading MIDAS database for sliced species 8 with 14 cores in total::start 1717128183.6: Downloading MIDAS database for sliced species 1 with 14 cores in total::start

Traceback (most recent call last): File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/main.py", line 29, in main() File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/main.py", line 25, in main return subcommand_main(subcommand_args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 148, in main download_midasdb(args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 37, in download_midasdb download_midasdb_worker(args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 91, in download_midasdb_worker midasdb.fetch_files("pangenome", species_id_list) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 190, in fetch_files return self.fetch_tarball(filename, list_of_species) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 215, in fetch_tarball md5_fetched = file_md5sum(_fetched_file) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 364, in file_md5sum return md5(open(local_file, "rb").read()).hexdigest() FileNotFoundError: [Errno 2] No such file or directory: '/Disk2/database/midas2/uhgg/pangenomes/100003/genes_info.tsv' 1717129477.5: Deleting untrustworthy dowdloaded databases. Specify --debug flag to keep. Traceback (most recent call last): File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/main.py", line 29, in main() File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/main.py", line 25, in main return subcommand_main(subcommand_args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 148, in main download_midasdb(args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 37, in download_midasdb download_midasdb_worker(args) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/subcommands/database.py", line 91, in download_midasdb_worker midasdb.fetch_files("pangenome", species_id_list) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 190, in fetch_files return self.fetch_tarball(filename, list_of_species) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 215, in fetch_tarball md5_fetched = file_md5sum(_fetched_file) File "/Disk1/mambaforge/envs/midas2.0/lib/python3.9/site-packages/midas2/models/midasdb.py", line 364, in file_md5sum return md5(open(local_file, "rb").read()).hexdigest() FileNotFoundError: [Errno 2] No such file or directory: '/Disk2/database/midas2/uhgg/pangenomes/100012/genes_info.tsv' 1717129492.7: Deleting untrustworthy dowdloaded databases. Specify --debug flag to keep.

yejunbin commented 3 months ago

The first error do not happened in MIDAS v2.0.1

midas2 database --init --midasdb_name uhgg --midasdb_dir /Disk2/database/midas2/uhgg 1717139679.6: Initate MIDASDB uhgg to /Disk2/database/midas2/uhgg::start 1717139740.7: Initate MIDASDB uhgg to /Disk2/database/midas2/uhgg::finish 1717139740.7: Download chunks for MIDASDB uhgg to /Disk2/database/midas2/uhgg::start 1717139831.6: Download chunks for MIDASDB uhgg to /Disk2/database/midas2/uhgg::finish

zhaoc1 commented 3 months ago

Hi,

I am in the middle of publishing the new MIDASDB with the latest version of MIDAS2, and that's the reason why you saw the error. It should be finished within the next month. The new release is mainly updates about the pangenome database. In the mean time, you can use the release v1.0.9.

Thanks, Chunyu

zhaoc1 commented 2 months ago

Hi, I just published the new MIADS version 3 code base as well as the updated MIDASDB. Please let me know if you encounter any issue. Thank you for your patience.

Chunyu