search

robbiebarrat / art-DCGAN

Modified implementation of DCGAN focused on generative art. Includes pre-trained models for landscapes, nude-portraits, and others.
Other
2k stars 340 forks source link

genre_scraper.py only scraping 4000 images max #18

Open toemm opened 5 years ago

toemm commented 5 years ago

The scraper works fantastic but is unable to get more than 3000-4000 images from wikiart. I tried adjustung num_pages (up to 4000 pages) but it won't scrape more than 4k pictures.

wikiart

Maybe it is because on the webpage it is also only showing max 3600 pictures? As can be seen here: https://www.wikiart.org/en/paintings-by-genre/portrait?select=featured#!#filterName:featured,viewType:masonry

Is there any fix to this because I'd like to train the network on more than 4k pictures.

robbiebarrat commented 5 years ago

try it now - i just updated the scraper

toemm commented 5 years ago

Thanks for the quick update but it still only attempts to load 3915 pictures, tried with different num_pages values but no avail.

wikiart2

toemm commented 5 years ago

I've tried everything and couldn't fix this. :( You updated the code but only changed one import, I don't think it does anything. The sites only shows 3600 pictures per style/genre.

robbiebarrat commented 5 years ago

I'll look into this more over the weekend - really sorry it doesn't work, and thanks for bringing it to my attention - leaving this thread open until i fix it...

josh-marsh commented 5 years ago

I am having the same problem. If this is not resolvable, would it be possible for you to upload the complete set of images that I assume you still have stored somewhere to a google drive folder? It would be incredibly appreciated. Cheers

robbiebarrat commented 5 years ago

@john-marsh i'm still looking into it - i think it might be a question of too many threads working at once... i think it will be resolvable.

josh-marsh commented 5 years ago

Cheers mate. I would try to fix it myself, but web scraping is not something that I have experience with. Keep us updated!

enochkan commented 5 years ago

any updates?

jyu-theartofml commented 4 years ago

Not sure if this is a related issue, but I had problem scraping image names that are not utf-8 compatible because it had accent characters. I fixed the problem by adding urllib.parse.quote under def downloader as follows,

file=urllib.parse.quote(file, safe=':/')
filepath = file.split('/')