Open wrmthorne opened 1 year ago
Is there any update on this?
As I never got a response, I assumed it was not something that was really wanted or was too small a feature to spend development time and effort on. I have been too occupied with other work since making this feature request so I haven't gotten around to it. If I ever implement it, I'll update here.
Description & Motivation
Description: A mode for combined_loader which would evenly distribute samples from all iterables.
Motivation: While working with cycle consistency training, two models are updated iteratively on the output of the other model. The current modes for combined_loader do not train these models optimally:
min_size: Wastes valuable data from larger dataset max_size: One model is trained on the static state of the other, causing it to overfit to an old model state max_size_cycle: May cause overfitting on smaller dataset if dataset sizes are significantly different sequential: Same problem as max_size but worse
Pitch
Interleaved mode which calculates how to evenly distribute one dataset with the other. This comes with the caveat of knowing the lengths of each iterable which may not be possible.
e.g.
Alternatives
No response
Additional context
I don't believe the
__len__
issue would be too damaging as pytorch dataloaders implement a__len__
method. Obtaining iterable lengths will get stuck on infinite iterables but lengths could otherwise be obtained with:cc @borda