Open iamwangyabin opened 1 year ago
thanks for this PR, I think I should refactor my code for better style
I have to mention that when I try to scrape thousands of images, this code often collapses with an SSL error. Specifically, the error message is raised as follows:
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='image.civitai.com', port=443): Max retries exceeded with url: /xG1nkqKTMzGDvpLrqFT7WA/8b41b365-e216-45c7-51a8-4278510f4900/width=1080/8b41b365-e216-45c7-51a8-4278510f4900.jpeg (Caused by SSLError(OSError(24, 'Too many open files')))
I add codes using this project and civitai's API to automatically scrape images from Civitai. I think it might be useful for others.