search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.87k stars 976 forks source link

Redgif User full gallery download (NSFW) #4583

Closed ciabello closed 1 year ago

ciabello commented 1 year ago

Hi all, i'm using the standalone version because i'm a noob, i'm trying to download all the gifs from a redgif user, so the basic command is the following:

C:\Users\PC> gallery-dl https://www.redgifs.com/users/ozyman

this downloads all the gifs that can be seen from the trending (or top, i haven't found out which one it is) page of the user, while the page is seen from me as "newest".

Problem is that the uploader has more uploads than the number of gifs loaded on the page (that can be verified by scrolling to the bottom when sorting from newest https://www.redgifs.com/users/ozyman?order=new and then seeing the 1st post on https://www.redgifs.com/users/ozyman?order=old).

also the downloader does not support the URL with the sorting URLs (i would have used it to circumvent the limitation on the number of posts).

Hrxn commented 1 year ago

Huh? Yes, it should download all "gifs" from an account, according to your linked profile page it should be 1808, as of writing this right now. How much do you got?

The only issue here is gifs that get deleted later, in most cases the total count accurately reflects that, but there is also some kind of "silent deletion" where the gif link completely disappears instead of the usual 404 error page, but this is very rare. I've only seen it happening for one or two redgif users, and this was very likely due to "problematic" content, although I'm just guessing here.

For the majority of all profiles, the gif count number should be correct, and all is downloaded as expected.

The only thing to be aware of, since gallery-dl's redgifs.py extracts info via the site API, is that (due to how their API works) the results (and therefore your downloads) appear in an order does not really make sense.

The same for your URL examples with ...?order=X, this changes the order of the gifs displayed on the page grid, but how is this relevant for downloading?

ciabello commented 1 year ago

I cited the order of the posts because it's the way i noticed there are files missing from the download: the last trending video is the last video that was downloaded. As for the number of files i got is 1000 -a weirldy round number-
the most recent (aka last modified) downloaded file is from 25 august, the most recent video on the channel is from 24 september, so there are files missing. I probably didn't explain properly before, i hope it's clearer now

Hrxn commented 1 year ago

Well, you might be onto something here.. something with this profile is fishy.

Is this account especially notorious, or went viral or something?

Not sure what to make of it.. But I've also found a user profile where this is exactly reversed (https://www.redgifs.com/users/miccapietro). It says 53 posts, I've downloaded 53 posts with gallery-dl, although I only ever seem to get 40 posts in the browser.

Hrxn commented 1 year ago

For the record, it is normal that you do not see all posts on a profile page. This is called lazy loading, and is a very common technique on many websites. As you said, you have to scroll to the bottom to keep loading more.

You know what, I have a somewhat old bit of JavaScript that I can adapt to Redgifs, to get the links directly from the profile page as seen in the browser. Let's see how many I actually get from your initial example link.

I will post the result in here.

Hrxn commented 1 year ago

Haha... guess this has to wait.

I'm only ever getting 40 posts as well from your example link. So, either my browser is broken, or the site is really acting up right now.

KevinKevinson5 commented 1 year ago

I think it might need a 'client' oauth token, the redgif api docs say that the default guest token has a low request limit.

https://github.com/Redgifs/api/wiki/API-access

I've tried downloading from several accounts and it successfully gets everything from any account with less than 80 video posts. Any more than that and it cuts off. The most I got was about 230 but it still added up to 80 if you count the images to videos in a 4:1 ratio, I got 200 images and 30 videos.

Hrxn commented 1 year ago

Okay, so it's not just me. Good to know.

Guess there has been a site change, has to be very recent, because not long ago I've downloaded some accounts with some 600, 700, 800 posts.

KevinKevinson5 commented 1 year ago

Something changed and it just downloaded over 700 videos from a single account without issue.

Hrxn commented 1 year ago

Some (mildly) interesting news.

The example link from OP is now working again in the browser, and is lazily loading posts again, instead of cutting off at 40.

Hrxn commented 1 year ago

Tried my link copying script bit.

Still ended with 1000 posts. So not all the profile count claims. Maybe the same 1000 @ciabello said?

mikf commented 1 year ago

Sounds like redgifs changed some internals and broke its site while doing so. I think I could only view the first 40 or so posts on the website and gallery-dl got 80.

Still ended with 1000 posts.

Maybe redgifs limits its pagination results to 1000? If so, I don't think there's much that can be done.

Support for ?order=... got added in https://github.com/mikf/gallery-dl/commit/be17103e21fd1c58575bf66bd7a8ac651889354d, by the way.