Improve metadata script

EBI-Metagenomics / genomes-catalogue-pipeline

MGnify genome analysis pipeline

Other

101 stars 21 forks source link

Improve metadata script #38

Closed KateSakharova closed 1 year ago

KateSakharova commented 1 year ago

Metadata script does a lot of ENA API calls to get information for each genome. If ENA API is unstable - whole script fails -> pipeline fails. -resume function runs everything from scratch. Maybe it makes sense to improve this script:

create a backup file for already fetched metadata
fetch as one big file and parse it
fetch metadata in the start of pipeline (in parallel with fetch ...)

mberacochea commented 1 year ago

Moved to -> https://www.ebi.ac.uk/panda/jira/browse/EMG-4408