mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

[Reddit] Allow downloading from quarantined subreddits #2180

Closed Unit193 closed 2 years ago

Unit193 commented 2 years ago

Howdy,

Seems yt-dlp in commit f0e5366 added the ability to download from quarantined subreddits, this could be useful in gallery-dl too.

This worked for a test, but isn't technically correct.

--- a/gallery_dl/extractor/reddit.py
+++ b/gallery_dl/extractor/reddit.py
@@ -19,7 +19,7 @@
     directory_fmt = ("{category}", "{subreddit}")
     filename_fmt = "{id}{num:? //>02} {title[:220]}.{extension}"
     archive_fmt = "{filename}"
-    cookiedomain = None
+    cookiedomain = ".reddit.com"

     def items(self):
         self.api = RedditAPI(self)
@@ -163,6 +163,7 @@
         if sub:
             self.subcategory += "-" + sub
         RedditExtractor.__init__(self, match)
+        self.session.cookies.set('_options', '%7B%22pref_quarantine_optin%22%3A%20true%7D')

     def submissions(self):
         return self.api.submissions_subreddit(self.subreddit, self.params)
@@ -188,6 +189,7 @@
         if sub:
             self.subcategory += "-" + sub
         RedditExtractor.__init__(self, match)
+        self.session.cookies.set('_options', '%7B%22pref_quarantine_optin%22%3A%20true%7D')

     def submissions(self):
         return self.api.submissions_user(self.user, self.params)
mikf commented 2 years ago

I tried this with the link from the yt-dlp issue and I'm only getting 403 error ({"message": "Forbidden", "error": 403}). I've also used the unquoted version {"pref_quarantine_optin": true} as cookie value, changed the domain for API requests from oauth.reddit.com to www.reddit.com, used public/private access tokens and no difference.

Unit193 commented 2 years ago

Checking that, seems the community was banned, so it no longer exists. I went down this list looking for something moderately innocuous and seems r/ApolloMoonLandingHoax still works with the example changes above. It's interesting to note though, while that pulls up fine, /r/DrawPeople (no idea what's on here, view at your own risk) requires one to log in, so gallery-dl gives [reddit][error] AuthorizationError: Insufficient privileges to access the specified resourc whereas the former subreddit worked fine.

Example output blow.

unit193@Sigma:~$ gallery-dl -s https://reddit.com/r/ApolloMoonLandingHoax
nl7x6a It's more that an educated person is more likely to make decisions base…stening to whatever furious white man is shouting at them on the radio, tel.png
n8o4k0 The Litmus Test For Stupid.png
msqo41 A Short Story.jpg
mrprwr Just that idea that some people will actually debate with you the finer…oints of the theoretical physics of this science fiction is just... idk smh.png
mm4yfk ‘We’re Apollo 13-ing this vaccine rollout’.png
mjl3pn I use my physics and calculus on a daily basis for engineering..png
mf1sod Is there anything I, or anyone, could say to change your mind?.png
m93p5u Another brave soul takes a swing at the 24 Apollo Moon Landing Questions..png
k9enam No shadow from flagpole on moon.jpg
k9eje2 Shadow of flagpole on moon.jpg
k2wtan wHaT iS uP?.jpg
g6wakm MIT proved no retroreflector necessary in 1962.png
e1pufc I can tell you that common sense doesn't belong in physics. You never go with your gut in physics because it is probably wrong.png
e1llws A Gut Feeling.png
dvkr2y Long Shadows On The Beach.png
dn2p57 I’m a physicist, so I understand!.jpg
dkqwoi What screams "I'm very insecure"?.png
b492q6 Moon Landing Hoax. Note that the there are astronaut footprints visible…some places, but no foot prints coming back from the LRV final parking spot.jpg
anxddy The Challenger Crew, 30 Years On ?.jpg
9hb8rw Q post 2224 - Q claims moon landings are real.png
9a4loo This is what United States Space Force posted to Twitter tonight. What do you think? Grab your popcorn!.jpg
96x9dg NASA ~ Major Story the Mainstream media decides to be silent once again… choose to ignore all the Pedos arrest by the DOJ.. these media people are .jpg
imgur_K3pn4E3_001_i1Igbzf.png
96fo5r Tell us again why you believe White Men walked on the moon? Be specific. Like the exact video or photograph or text that convinced you.png
95ah8x Why do you believe Americans walked on the Moon?.png
8rk2nz The State of Science: If you print enough money, eventually you can turn lead into gold, and have men walk on the moon.png
8rjfnv "The first gulp from the glass of natural sciences will make you an ath…ass God is waiting for you" ~ Werner Heisenberg (Father of Quantum Physics).jpg
8r054b 24 Questions. Apollo Moon Landing Hoax Story Problems for Math and Science Dummies #NASA #Panon #Qanon #Q.png
8pidv4 My theory is that you know absolutely nothing about space travel or rocket ships at all that you haven't seen in a science fiction movie.png
imgur_0uD7FsB_The liftoff of Apollo 15 from the surface of the moon.mp4
8p8jhm Apollo Lunar Lander isn't held together with "tape"... its held togethe…e strips"... That's not "black construction paper", its a "thermal coating".png
8p8aza Apollo Moon Landing Story Problems For Math and Science Dummies | #MoonLandingHoax #24Questions.png
8p85g9 user banned from _r_AskPhysics for asking, "If an astronaut is on Earth… have to go 'up' against the force of Earth's gravity to get to the Moon ?".png
8p84km apparently asking _r_AskPhysics if an astronaut on Earth must go 'up' a…nsidered spamming conspiracy theories and behavior worthy of the ban hammer.png
8oywy5 Apollo 17 Liftoff from Moon Dec 14, 1972 Note the Red, Green and Blue Pyrotechnics used for Special Effects.png
8oyo7u Apollo 17 Liftoff from Moon Dec 14, 1972 [L] Note white light at center…ase is now gone, and if light was supposedly flames from rocket exhaust, no.png
8oyk7j Apollo Lunar Lander Launch Hoax - Note the bright white center of the b…he rocket exhaust, then we can clearly see there is no exhaust coming from .png
8oyj32 Apollo Lunar Lander Launch Hoax - Note the bright white center of the b…upposedly flames from the rocket exhaust. The colored dots are pyrotechnics.png
8oi6qq The Lunar Lander is held together with tape.jpg
upload.wikimedia.org_wikipedia_commons_thumb_e_e4_Cernan_s71-51308.jpg_220px-Cernan_s71-51308.jpg
8oaje7 The Laughable Lunar Lander with Shiny Gold Foil.jpg
8oai4p Astronaut Buzz Aldrin walks on surface of moon near leg of lunar module…mmander Neil Armstrong took this photograph with 70mm lunar surface camera..jpg
8oaans Nothing "finally convinced" me. It is obviously true..png
8o9m2t Proof of Faked Moon Landing in 5 Seconds... the "Lander" shadow almost reaches the "horizon"!.jpg
8o3zgw Burden of Proof: "You need to prove people have not walked on the moon".png
8nyyxi Let your eye find the star closest to the center of the flag. Note that…the shape of the shadow in both flags is identical, and covers the exact sa.png
8nyw4v Let your eye find the star closest to the center of the flag. Note that…the shape of the shadow in both flags is identical, and covers the exact sa.jpg
mikf commented 2 years ago

So the problem was me testing with an already deleted subreddit ...

I added the cookie for public access tokens in https://github.com/mikf/gallery-dl/commit/2d34d8ff8bbec102358ef89ef5732fcb7afa0d25. According to my tests, it has no effect when using a private token. In that case you need to visit the quarantined subreddit with your account and agree to "view this community" there.

Unit193 commented 2 years ago

Sweet, thanks!