search

Imageomics / distributed-downloader

MPI-based distributed downloading tool for retrieving data from diverse domains.
MIT License
2 stars 0 forks source link

Bug in schedule creation #17

Open Andrey170170 opened 3 months ago

Andrey170170 commented 3 months ago

The schedule creation script doesn't properly use already downloaded and completed servers, creating more schedules than it actually requires. Verifier and Downloader on already downloaded schedules work properly - quickly completing the job. For example: _config.csv

index ServerName TotalBatches ProcessPerNode Nodes
16 www.jacq.org 8 1 1
23 ph.ansp.org 5 1 1
24 mcgillherbarium.com 1 1 1
25 media.api.aucklandmuseum.com 8 1 1
28 www.ucm.es 1 1 1

Fully consists of already downloaded servers.