Since the download can take a long time, is it possible to parallelize it across multiple jobs without SLURM? E.g. an option in the config or launch args to specify the total number of jobs and the ID of the current job. video2dataset ... --shard_id 0 --total_shards 100
habibian
commented
1 month ago
Hi,
Since the download can take a long time, is it possible to parallelize it across multiple jobs without SLURM? E.g. an option in the config or launch args to specify the total number of jobs and the ID of the current job.
video2dataset ... --shard_id 0 --total_shards 100