search

wix-incubator / DeviantArt-API

The DeviantArt API
https://www.deviantart.com/developers/
MIT License
35 stars 5 forks source link

can't access some images in very large (>10k) galleries #224

Closed colin-heberling closed 1 month ago

colin-heberling commented 11 months ago

When I make API requests to download from very large galleries (such as this one: https://www.deviantart.com/icedragon4u/gallery/all) the API doesn't return any more download links after ~9.5k-10k most recent image uploads to the gallery. If an artist organizes images into different folders, that can mitigate the issue, but often times they may have many images that are not organized into any folders, and are only accessible through the entire gallery. In this case, with the current API bug, it may make many or even the majority of an artist's gallery inaccessible through the API. The particular artist I have linked has also told me that they have already reached their folder limit, so asking them to put everything in folders is not an option, unless the folder limit is extended. Unless this was somehow intentional, this seems like a bug.

Could you look into fixing this?

sasha-lerner commented 1 month ago

this is an intentional limit

colin-heberling commented 1 month ago

Why? With AI art galleries, this threshold will be increasingly surpassed more and more often. That limit doesn't make any sense.

sasha-lerner commented 1 month ago

you can use endpoint '/gallery/{folderid}' as a workaround

colin-heberling commented 1 month ago

Did you read my opening message? I've already tried that as a workaround, and it doesn't always work, because the artist may have images that are not organized in folders, and there also appears to be a limit on how many folders an artist can put on their profile, so that's not a suitable solution.

You still didn't answer my question: why is there an intentional limit on how many images that can be downloaded from a single gallery to begin with? What is the purpose of that? There's already quite a bit of rate limiting, and downloading from multiple artists can easily download significantly more images than the limit imposed for a single artist, so if you say the reason is for individual user bandwidth limiting, then that reason doesn't fly.

As far as I can tell, it's an arbitrary, outdated limit, that should be updated, unless you can tell me otherwise. If you know that it's an intentional limit, then seemingly it should be very easy to update.