Add with_rank to Dataset.from_generator similar to Dataset.map and Dataset.filter.
Motivation
As for Dataset.map and Dataset.filter, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in the gen_kwars argument; however, this, in turn, includes the rank when computing the fingerprint.
Your contribution
Added #7199 which passes rank based on the job_id set by num_proc.
Feature request
Add
with_ranktoDataset.from_generatorsimilar toDataset.mapandDataset.filter.Motivation
As for
Dataset.mapandDataset.filter, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in thegen_kwarsargument; however, this, in turn, includes the rank when computing the fingerprint.Your contribution
Added #7199 which passes rank based on the
job_idset bynum_proc.