search

KuoriaNeko / civitai-scraper

A CivitAI scraper
GNU General Public License v3.0
36 stars 11 forks source link

add scrape images from Civitai #8

Open iamwangyabin opened 1 year ago

iamwangyabin commented 1 year ago

I add codes using this project and civitai's API to automatically scrape images from Civitai. I think it might be useful for others.

KuoriaNeko commented 1 year ago

thanks for this PR, I think I should refactor my code for better style

iamwangyabin commented 1 year ago

I have to mention that when I try to scrape thousands of images, this code often collapses with an SSL error. Specifically, the error message is raised as follows:

    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='image.civitai.com', port=443): Max retries exceeded with url: /xG1nkqKTMzGDvpLrqFT7WA/8b41b365-e216-45c7-51a8-4278510f4900/width=1080/8b41b365-e216-45c7-51a8-4278510f4900.jpeg (Caused by SSLError(OSError(24, 'Too many open files')))