search

beaufour / flickr-download

Tool for downloading sets and photos from Flickr
Apache License 2.0
231 stars 65 forks source link

Users list of set is up to 500 entries #58

Closed Cholaloula closed 2 years ago

Cholaloula commented 4 years ago

Hello,

I tried the option -l on a user in order to retake a download of all datasets broken (due to errornetwork) and made a list of all a user's set. I extracted the list but opening it I see the max entries is 500 when this user has clearly more

In this case it's this user: https://www.flickr.com/photos/biodivlibrary/

Is it a normal limitation due to API or something could be done retaining let's say the last ID of set and launching again the query?

I did it as my downloading of sets usually stops around 450, I don't know if it can be related but in this case the result of the command is the one expected.

Thanks for everything : )))

beaufour commented 4 years ago

Hmm, there is probably some pagination going on in the API... or there's a bug in the old version of the Flickr API I'm using. I'll look into it, although I cannot promise an ETA

Cholaloula commented 4 years ago

No problem, I'm trying homemade ways to get my goal, it was for informative purposes I highlighted it. It's the equivalent of 5 pages of albums results.

Thanks for all : )

eggplantedd commented 2 years ago

@beaufour I would also like to bump this, more important than the file names issue.

The 500 photoset limit means, in my example, out of 31,987 photos I can only get 2765.

I haven't tried downloading via -p because it would mean the pictures won't be organized into folders.

I am using the 0.7.3 flickr api.

beaufour commented 2 years ago

~Do any of you have an account with more than 500 public sets, so I can test this? I don't have that many sets myself...~

I see it in the original post... doh. Thanks :)

beaufour commented 2 years ago

this should be fixed on current master. Can you verify that it works for you?

eggplantedd commented 2 years ago

So just to be sure, I uninstalled the old flickr_download through PIP, then installed it again (showing new version number). I then checked the install folder to make sure none of my old files were lingering (they weren't).

Doing -l for photosets for biodivlibrary and alookback.net still showed me 500 listings max.

Did you break the 500 photoset limit for downloading, but not for listing? I will be testing it out now, but it will take a while to get to this stage so understandable why I went to -l first to see if it was fixed :)

beaufour commented 2 years ago

Oh sorry @eggplantedd , this is not in the released version. This is in master which is the current code here on github. Are you able to try that out?

beaufour commented 2 years ago

Otherwise, grab a "pre-release" here: http://beaufour.dk/flickr-download-0.3.1.tar.gz . You should be able to pip install flickr-download-0.3.1.tar.gz to get it installed.

beaufour commented 2 years ago

and the limit is removed from both listing and downloading

eggplantedd commented 2 years ago

Oh yes, no worries I will download it off the github. The flickr-api was pushed so got the special characters folder fix in just fine (although documenting list of changes in one txt might be useful, I could try and add that myself).

And yes- after downloading from master file and putting in the right %appdata% folder, it lists all the photosets right to the back! I shall be downloading now. Will get back to you if there's an API errors of which 31,987 photos should be a good test.

beaufour commented 2 years ago

Great to hear! 31k photos would be a pretty good test indeed :) The Flickr API is quite slow though, so it'll take quite a while...

eggplantedd commented 2 years ago

Yes- just had a program exit after a ERROR:root:Flickr error saving photo: HTTP Server Error 500: <!DOCTYPE html>

Had some of these before, but not always to end the program. There's no throttling of my connection either.

As it checks for existing files I will restart and continue on. The key has not expired as I've been able to restart on same one.

beaufour commented 2 years ago

Fixed in current master