Open phoenixdigital opened 3 years ago
I found the issue and preferred template after posting this so unfortunately didn't search for the problem first. There was a issue posted about Patreon but there wasn't enough info in the post to see if it was the same
https://github.com/ytdl-org/youtube-dl/issues/28671
whoops and another which is the same issue https://github.com/ytdl-org/youtube-dl/issues/26940
Hopefully my bug report shows a bit more detail about the issue at hand. Happy to do some testing or provide a more comprehensive HTML source if requested. I just won't post it publicly as it's full of PII.
After digging into it and the patreon.py code it looks like the module requires authentication now as this is the relevant section it errors on when not logged in and you can see it doesn't contain the "post_file" value which is visible when logged in.
"post": {
"data": {
"attributes": {
"change_visibility_at": null,
"comment_count": 168,
"created_at": "2021-04-13T08:30:22.000+00:00",
"current_user_can_comment": false,
"current_user_can_delete": false,
"current_user_can_view": false,
"deleted_at": null,
"edit_url": "/posts/tansjs-sunday-49973145/edit",
"edited_at": "2021-04-13T09:03:55.000+00:00",
"has_ti_violation": false,
"image": {
"height": 1080,
"url": "https://image.mux.com/eczqtdsUBnoy02rXLd8ldxfXmrSFCQWmGcKJiFq0202ruY/thumbnail.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ik5CY3o3Sk5RcUNmdDdWcmo5MWhra2lEY3Vyc2xtRGNmSU1oSFUzallZMDI0In0.eyJzdWIiOiJlY3pxdGRzVUJub3kwMnJYTGQ4bGR4ZlhtclNGQ1FXbUdjS0ppRnEwMjAycnVZIiwiZXhwIjoxNjIxMzc3NDA3LCJhdWQiOiJ0IiwidGltZSI6MTEyNy40fQ.SBLBfFgRfW8l2oei9F53VzDuXoJDcgV4D_FZtoV7vtTbr_QABtuZyVUGxM0j2eONDA-tfaUqkM4jjUIKbYUaRFZ0rOQj-qm0LN2BtG-tHTVY56iziRWxRz1UewltYXzggUM9eXnuqPCMpMgBguQ5lcHkE7u4zgA5Aa_s3n47-FRQWJTr6o_LbeTJD-IytWh4pPuwnJeUxsFvEs5lLVVX9NlbUPp5xA9Lm94BbDdmFFyuVRDq4yPS5EDtcDlYNxPVNuceDymVC4l6Cfn_KTdvoZ6OqVJJwY2XK2aWb2k49impL8nULTz20ZWVWbHoTKCm2tbIH1qXgHuXOH6pm9Vnaw",
"width": 1920
},
"is_automated_monthly_charge": false,
"is_paid": false,
"like_count": 157,
"meta_image_url": "https://c7.patreon.com/https%3A%2F%2Fwww.patreon.com%2F%2Fpost-teaser-image%2F49973145/selector/%23post-teaser",
"min_cents_pledged_to_view": 1,
"patreon_url": "/posts/tansjs-sunday-49973145",
"pledge_url": "/bePatron?patAmt=0.01\u0026c=1752462",
"post_metadata": null,
"post_type": "video_external_file",
"published_at": "2021-04-13T09:03:55.000+00:00",
"scheduled_for": null,
"teaser_text": null,
"title": "TANSJS: Sunday Magic (BETA TESTING)",
"upgrade_url": "/join/tellemstevedave/checkout?rid=2706415",
"url": "https://www.patreon.com/posts/tansjs-sunday-49973145",
"was_posted_by_campaign_owner": true
},
"id": "49973145",
"relationships": {
"access_rules": {
"data": [
{
"id": "365871",
"type": "access-rule"
}
]
},
"audio": {
"data": null
},
"campaign": {
"data": {
"id": "1752462",
"type": "campaign"
},
"links": {
"related": "https://www.patreon.com/api/campaigns/1752462"
}
},
"images": {
"data": []
},
"poll": {
"data": null
},
"ti_checks": {
"data": []
},
"user": {
"data": {
"id": "11190956",
"type": "user"
},
"links": {
"related": "https://www.patreon.com/api/user/11190956"
}
},
"user_defined_tags": {
"data": [
{
"id": "user_defined;All New Sunday Jeff Show",
"type": "post_tag"
},
{
"id": "user_defined;BETA",
"type": "post_tag"
},
{
"id": "user_defined;video",
"type": "post_tag"
}
]
}
},
"type": "post"
},
I tried uncommenting the login section of code in patreon.py but it didn't work. It's likely just a placeholder so I'll have to leave the rest to someone who knows a bit more about making a login module. Digging into the login process the URL is
https://www.patreon.com/api/login
It accepts these POST fields
{"data":{"type":"user","attributes":{"email":"theusername","password":"thepassword"},"relationships":{}}}
There is a form of 2FA where they ask you to approve that PC for login via email. So not sure what "fingerprint" it uses to detect a unique PC. Possibly this cookie in the request header to that login page.
cookie: __cfduid=_______redacted_____; patreon_device_id=_______redacted_____; patreon_location_country_code=AU; __cf_bm=_______redacted_____; _ALGOLIA=anonymous-_______redacted_____; _fbp=_______redacted_____; patreon_locale_code=en-US; G_ENABLED_IDPS=google; group_id=_______redacted_____; _swb=_______redacted_____
So you'd probably want to allow the user to pass in a patreon_device_id so they don't get forced with the 2FA everytime or cache it somewhere???
The response seems to set a cookie which might be used to keep the user logged in????
set-cookie: session_id=_____redacted______; Domain=patreon.com; Expires=Mon, 17-May-2021 22:--:00 GMT; Max-Age=2592000; Secure; HttpOnly; Path=/; SameSite=Lax
That's about all I could find. Hope that helps.
I bypassed the login code by supplying a cookies file, and I got a different error:
[Patreon] 54675251: Downloading JSON metadata
WARNING: "url" field is missing or empty - skipping format, there is an error in extractor
Traceback (most recent call last):
File "/usr/bin/youtube-dl", line 33, in <module>
sys.exit(load_entry_point('youtube-dl==2021.6.6', 'console_scripts', 'youtube-dl')())
File "/usr/lib/python3.9/site-packages/youtube_dl/__init__.py", line 475, in main
_real_main(argv)
File "/usr/lib/python3.9/site-packages/youtube_dl/__init__.py", line 465, in _real_main
retcode = ydl.download(all_urls)
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2068, in download
res = self.extract_info(
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 808, in extract_info
return self.__extract_info(url, ie, download, extra_info, process)
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 847, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 881, in process_ie_result
return self.process_video_result(ie_result, download=download)
File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1637, in process_video_result
if formats[0] is not info_dict:
IndexError: list index out of range
Two problems there, at least:
And then there's the problem of actually extracting valid link.
Also, if a page fails with yt-dl, it would be useful to know whether it works with yt-dlp, as that has a newer version of the extractor (but using some core functions not yet back-ported).
Checklist
Verbose log
Description
I was trying to download a video to keep locally for storage on my media server for personal use and got this error that asked me to post here.
youtube-dl-error.txt
and the relevant section of the HTML which I belive is related to the error is attached here. The whole file contains far too much pii to post publicly.
youtubedl-test.txt