Open NivekT opened 2 years ago
IIRC, we should enforce shuffling before sharding. cc @NicolasHug
If datasource is already shuffled to some extent, shard before shuffle might be valid operation. So warning IMO suffice.
We should add one more linter to check DataPipe object size and warn if it is too big (ex premature initialization of large structures).
shuffle
before each sharding
. This should mainly be used by domain libraries to verify their implementation. Note: users can still disable shuffle using DL2
For user-facing linter, we might provide a variant from this linter function. Raise warning when no shuffle
before sharding
and raise Error/warning if there is a shuffle
behind `sharding.
[ ] Warn if filter
appears between on_disk_cache
and end_caching
sections.
🚀 The feature
This issue proposes the addition of a linter for DataPipes and DataLoader2. The linter can analyze the graph of DataPipes and input arguments to DataLoaderV, and inform the users if any errors may occur ahead of time. The incomplete list of issues that the linter may try to analyze and raise is below. Please feel free to edit the list directly to add more or comment below.
Essential:
fork
fork
and etc)reset
the DataPipe in DataLoader)Nice-to-have:
fork
datapipe creates two or more copies ofStreamWrapper
orIOBase
Motivation, pitch
Having a linter will encourage best practices of DataPipe usages and reduces the number of unexpected bugs/behaviors in the data loading process during runtime.
Alternatives
Only raise exceptions during runtime.
Additional context
This linter is expected to work with DataPipes and DataLoaderV2. We should consider if it should work with the original DataLoader as well (and how).
cc: @VitalyFedyunin @ejguan