Is there a reason why if you specify 500 images per class, you get slightly more than 500

mf1024 / ImageNet-Datasets-Downloader

ImageNet dataset downloader. Creates a custom dataset by specifying the required number of classes and images in a class.

500 stars 131 forks source link

Open CMCDragonkai opened 4 years ago

CMCDragonkai commented 4 years ago

Running

python ./downloader.py -data_root ./tmp -number_of_classes 1000 -images_per_class 500

This in classes with 507 images, 505 images, 507 images... etc. What's the reason for this slight more than 500 images?

Does this have something to do with the concurrent downloading, and the fact that some URLs no longer work?

CMCDragonkai commented 4 years ago

Also I read https://mf1024.github.io/2019/06/09/how-to-scrape-the-imagenet/ which is great research on this issue. Imagenet is not really a reliable source of data.

However nowhere does it indicate why there slightly more images per class than requested.

huntkao commented 3 years ago

Does anybody know ? thanks.

ogkalu2 commented 1 year ago

@huntkao @CMCDragonkai It's the multiprocessing workers. It's fucking with the code. Set it to 1 and it'll download only as many as you specify