search

aditya-grover / climate-learn

Source code for ClimateLearn
MIT License
302 stars 49 forks source link

Deterministic randomness in ShardDataset #89

Closed prakhar6sharma closed 1 year ago

prakhar6sharma commented 1 year ago

Is your feature request related to a problem? Please describe. Currently ShardDataset, implements __iter__() to build batch. Inside the iter, the order of data is determined by self.epoch. Unfortunately, the self.epoch is incremented only for the child process and not for the parent process as a result, each epoch thus results in same shuffling order.

Describe the solution you'd like Having access to the trainer to retrieve epoch number or set up communication across child processes under the __iter__().

jasonjewik commented 1 year ago

Could we do something using the Lightning Module hooks like on_train_epoch_start?