search

rom1504 / img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
MIT License
3.71k stars 338 forks source link

Download Laion-coco lots of failed #439

Open LIRENDA621 opened 2 weeks ago

LIRENDA621 commented 2 weeks ago

When I download the laion-coco dataset via the url, about half of the downloads fail. Is this normal? Also, I want to know what download failure and resize failure mean

image
rom1504 commented 2 weeks ago

You can enable wandb or read the json files in output folder to find the precise error failure

Usual issue is not setting up a good dns resolver (see readme)

50% seems low. 75% success may be normal

On Sun, Nov 3, 2024, 06:28 LIRENDA621 @.***> wrote:

When I download the laion-coco dataset via the url, about half of the downloads fail. Is this normal? Also, I want to know what download failure and resize failure mean image.png (view on web) https://github.com/user-attachments/assets/3dddbb98-1019-4e28-9aab-9b580c6ef032

— Reply to this email directly, view it on GitHub https://github.com/rom1504/img2dataset/issues/439, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437WULEDD6PBVZUNBPKDZ6WX7JAVCNFSM6AAAAABRCMV52OVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTAOJYGE2TKOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>