search

tmbdev-archive / pytorch-imagenet-wds

25 stars 4 forks source link

Train and Valid dataset: WDS and Origin? #3

Open melgor opened 3 years ago

melgor commented 3 years ago

Hi, first of all, WDS is a great package, I have limited space on SSD, and thanks to WDS I can read the data from 4TB drive with a similar speed to SSD.

I have a question about the example here. Any reason why Train loader use WDS and Valida origin dataloader? I'm planing to use for both WDS. It was just for comparison purposes? Or there is some logic behind?

tmbdev commented 3 years ago

This repo is a minimal modification of the original PyTorch Imagenet example; its purpose is primarily to illustrate how easy it is to modify existing code. It doesn't represent best practices for actually using WebDataset.

So, for best practices, you would use WDS for both training and validation data.

I'll try to post a best practices example soon.