search

tcxxxx / DermNet-images-crawler

Web crawler for DermNet (http://www.dermnet.com/) - one of the greatest data resources for skin diseases.
21 stars 7 forks source link

Error in downloading few files #3

Open pumpkinband opened 6 years ago

pumpkinband commented 6 years ago

Hundreds of this Error :-

HTTPConnectionPool(host='www.dermnet.com', port=80): Max retries exceeded with url: /dn2/allJPG3/warts-treatment-2.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0cbab63fd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Failed to saved ./DermNet/Warts-Molluscum-and-other-Viral-Infections/Warts-Treatment/warts-treatment-2.jpg http://www.dermnet.com/dn2/allJPG3/warts-treatment-2.jpg

tcxxxx commented 6 years ago

Sorry for the late reply. I am checking out what's going on.

tcxxxx commented 6 years ago

Hundreds of this Error :-

HTTPConnectionPool(host='www.dermnet.com', port=80): Max retries exceeded with url: /dn2/allJPG3/warts-treatment-2.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0cbab63fd0>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Failed to saved ./DermNet/Warts-Molluscum-and-other-Viral-Infections/Warts-Treatment/warts-treatment-2.jpg http://www.dermnet.com/dn2/allJPG3/warts-treatment-2.jpg

I think this thread helps to understand the problem, and setting a time interval between requests probably helps: https://stackoverflow.com/questions/23013220/max-retries-exceeded-with-url