search

Imageomics / distributed-downloader

MPI-based distributed downloading tool for retrieving data from diverse domains.
MIT License
2 stars 0 forks source link

Parameterize preprocessing scripts #5

Open egrace479 opened 1 month ago

egrace479 commented 1 month ago

In reference to question on generalization of scripts/server_downloading_prep.slurm:

Yes, it'd need another bash script (e.g. something like submit_server_downloading_prep.sh) to submit this Slurm job with parameterized account info. I think parameterizing preprocessing scripts like this could be added as another issue.

_Originally posted by @thompsonmj in https://github.com/Imageomics/distributed-downloader/pull/4#discussion_r1617780675_

egrace479 commented 1 month ago

May also want to consider option to set batch size in .env as noted here.