search

wchen342 / SketchyGAN

Code for paper "SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis"
MIT License
131 stars 32 forks source link

Flickr crawler #12

Closed sitongsu closed 5 years ago

sitongsu commented 5 years ago

I've been trying crawling images by running flickr_crawler.py. But it occurs to have something wrong with it. 2018-10-31 18:16:19,205 - ERROR - parser - Exception caught when fetching page https://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=8dbdbcb0d6a 5820f28b4e7cdd24ab2c0&format=json&nojsoncallback=1&text=car&sort=relevance&per_pag e=50&min_upload_date=2017-11-20&max_upload_date=2018-04-21&page=1, error: HTTPSConnectionPool(host='api.flickr.com', port=443): Max retries exceeded with url: /serv ices/rest/?method=flickr.photos.search&api_key=8dbdbcb0d6a5820f28b4e7cdd24ab2c0&fo rmat=json&nojsoncallback=1&text=car&sort=relevance&per_page=50&min_upload_date=201 7-11-20&max_upload_date=2018-04-21&page=1 (Caused by NewConnectionError('<requests .packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe81b465f60>: Failed to establish a new connection: [Errno 111] Connection refused',)), remaining retry times: 2 So i set a proxy pool to deal with it.But it failed with the same error. I was wondering whether you have met such a problem and how do you deal with it? looking forward to your reply.thanks a lot.

YellowPig-zp commented 5 years ago

I encountered the issue from time to time, but it seems not to be a big problem because even though it failed to retrieve some images, most of the time it fetched the images successfully.

sitongsu commented 5 years ago

I encountered the issue from time to time, but it seems not to be a big problem because even though it failed to retrieve some images, most of the time it fetched the images successfully.

have you made any modification to the original code? or have you set proxies?

YellowPig-zp commented 5 years ago

I didn't modify the original code other than setting path to save the images and what categories I want to fetch.

sitongsu commented 5 years ago

I didn't modify the original code other than setting path to save the images and what categories I want to fetch.

do you mind sharing the images you fetched with me? i'd appreciate your help.thank you

wchen342 commented 5 years ago

I think it is because you make inquiries too fast to the server so it refuses your connections. Maybe lower the rate of querying?