search

tristandeleu / pytorch-meta

A collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch
https://tristandeleu.github.io/pytorch-meta/
MIT License
1.97k stars 256 forks source link

How does the multi-processing for meta-dataloader work? #71

Closed renesax14 closed 4 years ago

renesax14 commented 4 years ago

This is more out of curiosity! But I was curious to understand how you handle multiprocessing to make the pre-fetching of batches (tasks) in your meta-data loaders more efficient.

Are you using the standard pytorch dataloader to pre-fetch your meta batches/tasks? How is that being efficiently implemented so that the next time the loop is met the batch is pre-fetched and ready for training?

tristandeleu commented 4 years ago

The MetaDataLoader (and BatchMetaDataLoader) inherits from PyTorch's DataLoader: this really is just syntactic sugar to change the collate function and the samplers of DataLoader. The way data is loaded is exactly the same as in DataLoader, and so the way multiprocessing work is handled directly by PyTorch.