not downloading all the images

[zx96-001]

Hello, it's an exciting script, however, it doesn't download all the images from the users, in more than one instance I verified this. I'll leave just one example: user - betweenspectrums Even when running the script twice, it fails to archive many of the older images.

There are many on the page but none is archived, in the first series only number: 34920 gets downloaded.

The user has however also those: 34917, 34918, 34919, 34921, 34922, 34923, 34924, 34925, 34926, 34927, and the list goes on.

It's a great help but it's not working perfectly and for some reason, it skips entirely the images (and many more).

Could you maybe reproduce the same error and see if there is anything to change to let maybe it harvest a little slower but more completely?

[Confuzu]

Yes i could replicate the error but its not a problem of the script. The CivitAI API gives only 229 image on your example username betweenspectrums but he has 330 images uploaded.

My script works with what the API reports back. Its not scrapping the website it self. Sorry there is nothing i could change on my script to get them all. Sorry

But thanks for the hint, so I found out that the API does not spit out everything.

[zx96-001]

Thanks for the quick answer!

Could there be a workaround perhaps? I wonder what could be a way to scrape them all.

I am very thankful for your work anyway, but could be awesome to make another program to get them all.

Another small issue: if the prompts are not embedded in the jpg, but present in the description, we don't get them. I noticed this in various images of that same user, if I click the jpg there is no info, but on the website, if I check the same image, the prompt is there.

[Confuzu]

I have changed a few things and solved the problem with the meta data as far as it can be solved from the outside. readme update

[zx96-001]

fantastic, thanks so much!! :D