fail to download laion face image

KimWu1994 commented 1 year ago

hello, when I run

bash download.sh ./laion_face_meta ./laion_face_data

tha download fails

total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 258 - count: 20000
worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 131 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 387 - count: 30000
5it [01:23,  8.00s/it]worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 124 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 495 - count: 40000
worker  - success: 0.003 - failed to download: 0.997 - failed to resize: 0.000 - images per sec: 123 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 616 - count: 50000
9it [01:28,  2.50s/it]worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 116 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 698 - count: 60000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 116 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 811 - count: 70000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 117 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 927 - count: 80000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 117 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1043 - count: 90000

How can I fix it? Thanks.

2629630207 commented 1 year ago

Hello，did you solve the problem?

calcoloergosum commented 1 year ago

I've encountered the same issue, and partially resolved it.

laion_face_data/*/*_stats.json files reported many name resolution failure
- something like: <urlopen error [Errno -3] Temporary failure in name resolution>: 9950
While running download.sh, ping google.com reported ping: google.com: Temporary failure in name resolution.
a fast dns resolver didn't resolve the issue.

Lowering process count and thread count made it significantly better. (in download.sh, line 11, I changed --processes_count 16 --thread_count 128 to --processes_count 2 --thread_count 16

1it [09:09, 549.96s/it]worker  - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000
total   - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000
2it [09:18, 231.67s/it]worker  - success: 0.802 - failed to download: 0.189 - failed to resize: 0.009 - images per sec: 18 - count: 10000
total   - success: 0.807 - failed to download: 0.184 - failed to resize: 0.009 - images per sec: 36 - count: 20000

After the change, laion_face_data/*/*_stats.json looked sane.
80% of URL being up should be normal.
It may be the issue of my DNS server, but I stopped digging further.

2629630207 commented 1 year ago

Thank you very much, it helps me a lot

2629630207 commented 1 year ago

Thank you for your suggestion, it helps me a lot

------------------ 原始邮件 ------------------ 发件人: "Han @.>; 发送时间: 2023年8月27日(星期天) 凌晨0:45 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [FacePerceiver/LAION-Face] fail to download laion face image (Issue #10)

I experience the same issue.

a fast dns resolver didn't resolve the issue

Lowering process count and thread count made it significantly better. (in download.sh, line 11, I changed --processes_count 16 --thread_count 128 to --processes_count 2 --thread_count 16 1it [09:09, 549.96s/it]worker - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000 total - success: 0.813 - failed to download: 0.179 - failed to resize: 0.008 - images per sec: 18 - count: 10000 2it [09:18, 231.67s/it]worker - success: 0.802 - failed to download: 0.189 - failed to resize: 0.009 - images per sec: 18 - count: 10000 total - success: 0.807 - failed to download: 0.184 - failed to resize: 0.009 - images per sec: 36 - count: 20000
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

askerlee commented 9 months ago

Thanks for the tips. In my case, setting up a fast DNS server speeds up downloading significantly.

FacePerceiver / LAION-Face

fail to download laion face image #10