search

santosderek / Imgur-To-Folder

Download Imgur albums and images to desired folder
Other
80 stars 9 forks source link

Not able to download more than 3060 favorites at a time. #4

Open Sylvanas1000 opened 6 years ago

Sylvanas1000 commented 6 years ago

This is a great program, but it's not able to grab all of my favorites. For either the imgurToFolder --download-latest-favorites [username] or the imgurToFolder --download-oldest-favorites [username], it's only grabbing the latest or oldest 51 "pages" of imgur favorites each page containing 60 pictures or albums for a total of 3060 grabs per command. This is preventing me from getting all of my favorites. Any suggestions for a workaround or is this an innate issue with the imgur api?

santosderek commented 6 years ago

Thank you for letting me know! I know that the Imgur API supports up to 12,500 requests per day so you shouldn't be hitting that barrier unless you have more than 10,000 images, allowing 2500 for API calls.

I will implement a way to start at a certain "page", (something like itf --download-latest-favorites [username] --page-start 51) so that you should be able to pick up where you left off. Please allow me some time to implement this.

In the current implementation, when a user executes imgurToFolder --download-latest/oldest-favorites [username] it will create a list of all known favorites (at least as much as the api gives), and will skip over already downloaded images. So currently if you run the imgurtofolder command in the same directory as before, it should skip over already downloaded files, and will not use additional API calls / GET requests during this skip.

Also if you use imgurToFolder --list-all-favorites [username] | OR | imgurToFolder -lf [username], at the very top if should give you a number of known favorites that it was able to find. Is that number more that 3060?

Lastly, (sorry for the block of text), I will be implementing the --page-start feature in the next version. I will also figure out why it is stopping specifically at 51 pages.

I hope this helps! If there is something else I can improve or you have suggestions of what to implement let me know!

Sylvanas1000 commented 6 years ago

I don't think I've quite reached 10,000 images in my favorites yet, so I don't think that's the issue here.

When I run imgurToFolder --list-all-favorites [username] | OR | imgurToFolder -lf [username], it returns with 3060 favorites that it finds, although I have more, so like I said, I think this might be a cap with imgur's API. I appreciate your reply and look forward to seeing if you can resolve this issue in a future update.

Marooned-MB commented 1 year ago

Also if you use imgurToFolder --list-all-favorites [username] | OR | imgurToFolder -lf [username], at the very top if should give you a number of known favorites that it was able to find. Is that number more that 3060?

Unfortunately, imgurToFolder does not print to standard output so I can't do python imgurToFolder -lf username > list.json nor | more and the buffer limit in windows console is 9999 lines - this is not enough to see the top of that output so no way of telling what's there.

The program only download 396 Files, 28 Folders of my favs. Way less than said API limit and even less than 3600 mentioned in this ticket.

[edit] I was able to download more using --start-page param.