search

tristandeleu / pytorch-meta

A collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch
https://tristandeleu.github.io/pytorch-meta/
MIT License
1.97k stars 256 forks source link

How to add my own dataset to torchmeta #128

Open Balabala-Hong opened 3 years ago

Balabala-Hong commented 3 years ago

I want to conduct some experiments on other datasets using torchmeta, for example, NWPU-RESISC45. How to process the original file to make it well-matched with torchmeta (as I found the miniimagenet dataset is transformed and saved into JSON and HD5 files). I suggest you add some detailed descriptions about the content and the format of the standard dataset in the documentation. And the corresponding code for parsing and sampling the dataset is suggested (maybe just a simple template is enough). Because without these descriptions, the torchmeta can be kind of diffcult for beginners to learn.
Besides, the function check_integrity() in pytorch-meta/torchmeta/datasets/utils.py seems to lose effect in miniimagenet.py for unfound definition. Add from torchvision.datasets.utils import check_integrity in pytorch-meta/torchmeta/datasets/utils.py helps solve this bug.

AlaalaMo commented 3 years ago

I am also wondering and would be nice to know how to do this for a general dataset

tristandeleu commented 3 years ago

I'm sorry for the late reply. There is unfortunately no proper documentation to add your own dataset, but I have given a couple of guidelines here https://github.com/tristandeleu/pytorch-meta/issues/113#issuecomment-773972272 in order to get your own dataset to work with Torchmeta. My best advice is to look at how Omniglot works, with the 3 levels of abstractions required, to get inspiration and adapt it to your own dataset. I hope this helps.