Open MarqFJA87 opened 1 year ago
Can you isolate the specific post URL (i.e. like https://www.patreon.com/posts/precious-metal-23563293
), and see if this can still be reproduced this way?
No, unfortunately. It's buried deep within the list of files on the creator's page (the 860th post, counting from the most recent and not counting any of the Patron-exclusive posts since I'm not a Patron yet), and trying to manually browse through said list eventually results in a time-out about "halfway" through, forcing a page refresh.
I think I found a lead that could help solve this mystery. Three new posts have been added to this Patreon gallery over the past 3 days, so I moved my previous batch of downloads into a temporary folder and redid the mass-download of the gallery. Lo and behold, the exact same number of posts were downloaded as the previous batch (673), only now the download was interrupted at the post that's third from last in the original batch.
It seems that gallery-dl can only download 673 posts before hitting some sort of "wall" that prevents it from accessing any further posts.
It might be some sort of captcha like in https://github.com/mikf/gallery-dl/issues/3466#issuecomment-1381988225 that gets triggered after a certein number of requests.
You could try your luck with --sleep-request
and also check Patreon's responses with --write-pages
About --write-pages
... What does "write downloaded intermediary pages to files in the current directory" mean? Is it going to create additional text files for each downloaded file?
--write-pages
writes the (mostly JSON) data that gets send by Patreon to .txt files in the directory you run gallery-dl from. It is mostly useful for debugging purposes.
For example, running gallery-dl --write-pages https://www.patreon.com/posts/precious-metal-23563293
in /tmp/
creates two files containing the API and/or HTML response from Patreon
In your case, it would write the full content of the 403 response to disk, so you can take a better look at it and know what it actually was that Patreon complained about.
Okay, it's done. Attaching the generated files; 600+ were completely empty, so I've omitted them.
01_https_www.patreon.com_haganef_posts.txt 02_https_www.patreon.com_api_posts_include_campaign,access_rules,attachments,audio,images,media,native_video_insights,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.curre.txt 03_https_www.patreon.com_api_user_2941319.txt 18_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 32_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 48_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 63_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 76_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 92_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_responses.txt 108_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 123_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 136_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 152_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 169_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 183_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 198_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 213_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 227_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 243_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 259_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 275_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 290_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 305_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 320_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 336_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 351_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 367_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 382_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 398_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 413_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 427_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 443_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 457_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 473_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 486_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 503_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 517_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 532_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 546_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 562_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 576_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 590_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 601_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 611_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 626_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 640_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 652_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 666_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 678_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 689_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 700_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 712_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt 726_https_www.patreon.com_api_posts_include_campaign%2Caccess_rules%2Cattachments%2Caudio%2Cimages%2Cmedia%2Cnative_video_insights%2Cpoll.choices%2Cpoll.current_user_responses.user%2Cpoll.current_user_response.txt
The last file (726_...
) contains a (Cloudflare?) captcha page, most likely triggered by too many requests over time. As I said in https://github.com/mikf/gallery-dl/issues/3624#issuecomment-1424832312, try using --sleep-request
with some random interval like 1-3
to force some pauses between HTTP requests, or maybe just plain --sleep
.
Visiting the last URL that triggered the 403 in your browser (https://www.patreon.com/api/posts?include=...
), solving the captcha, and using exported cookies from that may also work.
Please enable JS and disable any ad blocker
```Nope, both commands run into the exact same problem at pretty much the exact same "distance" from the most recent post. And trying to open the last URL just gives me an error page with the following message.
{"errors":[{"code":3,"code_name":"ParameterInvalid","detail":"Invalid parameter for 'page[cursor]': Invalid or expired cursor.","id":"59850448-d132-5788-b4bf-8c1d2e88e375","source":{"parameter":"page[cursor]"},"status":"400","title":"Invalid value for parameter 'page[cursor]'."}]}
I'm trying to download https://www.patreon.com/haganef using cookies, and the process runs normally until it reaches a certain post, whereupon it gives an error and stops.
The verbose log starts with the following:
When it hits the error, I get the following: