vector-engineering / covidcg

A COVID-19 CoV Genetics (CG) browser to inform therapeutics development
https://covidcg.org
MIT License
26 stars 5 forks source link

Chunk GenBank (NCBI Virus vvsearch) downloads #591

Closed atc3 closed 1 year ago

atc3 commented 1 year ago

Current downloads for SARS2 on the NCBI Virus endpoint are timing out due to large data size. This PR chunks out downloads by a user-configurable period and combines chunks to feed into existing ingest workflows.