search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.93k stars 975 forks source link

[Patreon] Patreon videos attempted to download through ytdl (yt-dlp) receive error with lastest Workflow Executable #5885

Closed ghost closed 3 months ago

ghost commented 3 months ago

System: Windows 11 64bit Gallery-DL Version: executable-windows-latest-x64-3.12 (from [tumblr] implement 'pagination' option (#5880) #689 )

I tried with the release version, but that failed to download all together and the yt-dlp config file created error--probably because it wasn't a workflow executable and didn't have the yt-dlp module built in. However videos added as attachment without the patreon player active are downloaded.

PS C:\Apps\Tools - Utility\DL-Files> gallery-dl --verbose https://www.patreon.com/feedfancier/
[gallery-dl][debug] Version 1.27.2-dev:2024.07.23 - Executable (dev/windows)
[gallery-dl][debug] Python 3.12.4 - Windows-11-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.2
[gallery-dl][debug] Configuration Files ['C:\\Apps\\Tools - Utility\\DL-Files\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/feedfancier/'
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/feedfancier/'
[patreon][debug] TLS 1.2 disabled.
[patreon][debug] Loading cookies from 'C:/Apps/Tools - Utility/DL-Files/patreon-cookies.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /feedfancier/posts HTTP/11" 200 None
[patreon][debug] campaign_id: 104848
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=campaign,access_rules,attachments,audio,images,media,native_video_insights,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,user,user_defined_tags,ti_checks&fields%5Bcampaign%5D=currency,show_audio_post_download_links,avatar_photo_url,avatar_photo_image_urls,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Bpost%5D=change_visibility_at,comment_count,commenter_count,content,current_user_can_comment,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,insights_last_updated_at,is_paid,like_count,meta_image_url,min_cents_pledged_to_view,post_file,post_metadata,published_at,patreon_url,post_type,pledge_url,preview_asset_type,thumbnail,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner,has_ti_violation,moderation_status,post_level_suspension_removal_date,pls_one_liners_by_category,video_preview,view_count&fields%5Bpost_tag%5D=tag_type,value&fields%5Buser%5D=image_url,full_name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&fields%5Bmedia%5D=id,image_urls,download_url,metadata,file_name&fields%5Bnative_video_insights%5D=average_view_duration,average_view_pct,has_preview,id,last_updated_at,num_views,preview_views,video_duration&filter%5Bcampaign_id%5D=104848&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bis_draft%5D=false&sort=-published_at&json-api-version=1.0 HTTP/11" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/243223 HTTP/11" 200 None
[patreon][debug] Using download archive 'C:/Apps/Tools - Utility/DL-Files/archive/patreon_archive.sqlite3'
[patreon][debug] Active postprocessor modules: [MetadataPP]
[patreon][error] An unexpected error occurred: OptParseError - Usage: gallery-dl.exe [OPTIONS] URL [URL...]

gallery-dl.exe: error: no such option: -_
. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[patreon][debug]
Traceback (most recent call last):
  File "optparse.py", line 1387, in parse_args
  File "optparse.py", line 1431, in _process_args
  File "optparse.py", line 1513, in _process_short_opts
optparse.BadOptionError: no such option: -_

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gallery_dl\job.py", line 152, in run
  File "gallery_dl\job.py", line 195, in dispatch
  File "gallery_dl\job.py", line 329, in handle_url
  File "gallery_dl\job.py", line 495, in download
  File "gallery_dl\downloader\ytdl.py", line 50, in download
  File "gallery_dl\ytdl.py", line 43, in construct_YoutubeDL
  File "gallery_dl\ytdl.py", line 72, in parse_command_line
  File "yt_dlp\options.py", line 113, in parseOpts
  File "yt_dlp\utils\_utils.py", line 4953, in parse_args
  File "optparse.py", line 1389, in parse_args
  File "yt_dlp\options.py", line 204, in error
optparse.OptParseError: Usage: gallery-dl.exe [OPTIONS] URL [URL...]

gallery-dl.exe: error: no such option: -_
mikf commented 3 months ago

You appear to be passing -_ as a command-line option to yt-dlp, which isn't a valid one.

edit: gallery-dl.exe: error: no such option: -_ is actually yt-dlp: error: …, but OptParse uses the initial program name here.

ghost commented 3 months ago

@mikf Thank you for the reply.

I've looked through both my gallery-dl.conf and yt-dlp.conf. I can't see where I might be passing "-_" as an option. I also deleted all the content of the yt-dlp.conf and just ran it, and I received the same error message, so it would have to be in my gallery-dl.conf file, but I can't find anywhere that it might add "-_" under patreon, ytdl (both in extractor and downloader), or the main extractor arguments.

I am not sure if I am allowed to attach my config files, but I added gallery-dl.conf and yt-dlp.conf for you to look at (converted to txt because github won't allow me to upload a .conf file):
REDACTED & yt-dlp.txt

A few things to note, both configs are under the same folder 'DL-Files'. This contains gallery-dl.exe, yt-dlp.exe (I only recently learned gdl doesn't use the external executable), an ffmpeg.exe (probably outdated and needs replaced), and a few other executables, as well as text files for import lists, ps1/powershell files for easy batch running of the executables, and a folder for gdl archives. Perhaps that could be causing issue, but I've not had any problems before; the folder is added to my PATH environment so it's easier to add executables doing similar actions to the same folder.

The command I am running from PowerShell is literally just "gallery-dl \<link> [links]", since everything I need for cookies, directory structure, and the like is in the config file.

Finally, I've also tried to install gallery-dl and yt-dlp through python/pip and moved the python modules folder above my DL-Files folder in PATH so it sees that first and runs those instead. This didn't do anything--same error, and I made sure to include the path to my config file with "-c 'path/to/config'".

Thank you for taking a look and troubleshooting with me.

Edited: to add the escape character for certain characters that triggered markdown

mikf commented 3 months ago

I might have found the issue: You've set a cookies file as yt-dlp config file in your gallery-dl settings.

"config-file" : "C:/Apps/Tools - Utility/DL-Files/yt-cookies.txt",

I've also found a bug with how gallery-dl forwards --cookies-from-browser to yt-dlp while looking into this, so there's also that.

ghost commented 3 months ago

@mikf Thank you.

I changed the path to the config file for ytdl to the actual config--thank you for catching that--and it seems to have resolved that error. However, now I have a different error

[patreon][error] An unexpected error occurred: TypeError - expected string or bytes-like object, got 'dict'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .

I went on to try and disable the fotwarding of cookies, but that didn't resolve the error--I also have my patreon module using a cookies file instead of browser, so I am not sure if that overrides what would be sent to yt-dlp or not, but either way it doesn't seem to be the current issue. I kept that line turned to false just in case because I think my yt-dlp config loads cookies from browser anyway.

I am providing the new config file for gdl because I realized I had passwords for both Patreon and Bluesky in the one I uploaded above, so I want to disable that link: gallery-dl.txt

I noticed this popped up after the MetadataPP, so I also tried to delete or disable any lines that had to do with metadata, both in the yt-dlp config and gallery-dl config. This includes the metadata post processor found in the patreon extractor and the metadata line to false. This didn't seem to affect anything.

Below is the error output:

C:\Users\Ari-Wonderbelt>gallery-dl --verbose https://www.patreon.com/feedfancier/
[gallery-dl][debug] Version 1.27.2-dev:2024.07.23 - Executable (dev/windows)
[gallery-dl][debug] Python 3.12.4 - Windows-11-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.2
[gallery-dl][debug] Configuration Files ['C:\\Apps\\Tools - Utility\\DL-Files\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/feedfancier/'
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/feedfancier/'
[patreon][debug] TLS 1.2 disabled.
[patreon][debug] Loading cookies from 'C:/Apps/Tools - Utility/DL-Files/patreon-cookies.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /feedfancier/posts HTTP/11" 200 None
[patreon][debug] campaign_id: 104848
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=campaign,access_rules,attachments,audio,images,media,native_video_insights,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,user,user_defined_tags,ti_checks&fields%5Bcampaign%5D=currency,show_audio_post_download_links,avatar_photo_url,avatar_photo_image_urls,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Bpost%5D=change_visibility_at,comment_count,commenter_count,content,current_user_can_comment,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,insights_last_updated_at,is_paid,like_count,meta_image_url,min_cents_pledged_to_view,post_file,post_metadata,published_at,patreon_url,post_type,pledge_url,preview_asset_type,thumbnail,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner,has_ti_violation,moderation_status,post_level_suspension_removal_date,pls_one_liners_by_category,video_preview,view_count&fields%5Bpost_tag%5D=tag_type,value&fields%5Buser%5D=image_url,full_name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&fields%5Bmedia%5D=id,image_urls,download_url,metadata,file_name&fields%5Bnative_video_insights%5D=average_view_duration,average_view_pct,has_preview,id,last_updated_at,num_views,preview_views,video_duration&filter%5Bcampaign_id%5D=104848&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bis_draft%5D=false&sort=-published_at&json-api-version=1.0 HTTP/11" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/243223 HTTP/11" 200 None
[patreon][debug] Using download archive 'C:/Apps/Tools - Utility/DL-Files/archive/patreon_archive.sqlite3'
[patreon][error] An unexpected error occurred: TypeError - expected string or bytes-like object, got 'dict'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[patreon][debug]
Traceback (most recent call last):
  File "gallery_dl\job.py", line 152, in run
  File "gallery_dl\job.py", line 195, in dispatch
  File "gallery_dl\job.py", line 329, in handle_url
  File "gallery_dl\job.py", line 495, in download
  File "gallery_dl\downloader\ytdl.py", line 50, in download
  File "gallery_dl\ytdl.py", line 68, in construct_YoutubeDL
  File "yt_dlp\YoutubeDL.py", line 774, in __init__
  File "yt_dlp\YoutubeDL.py", line 2514, in build_format_selector
  File "re\__init__.py", line 186, in sub
TypeError: expected string or bytes-like object, got 'dict'
mikf commented 3 months ago

This error happens because you've put a format setting at the top level of your config, outside of output where it should be (It's at the very bottom of the file).

Options in this position count as global, so this overrides your ytdl format settings and causes an exception since it's the wrong type.

ghost commented 3 months ago

@mikf

I believe the format I was supposed to move was the one with the debug, info, warning, and error message settings. Tell me if I am wrong. gallery-dl.txt

That resolved the type error, but now I get this for output with all video posts:

C:\Users\Ari-Wonderbelt>gallery-dl --verbose https://www.patreon.com/feedfancier/
[gallery-dl][debug] Version 1.27.2-dev:2024.07.23 - Executable (dev/windows)
[gallery-dl][debug] Python 3.12.4 - Windows-11-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.2.2
[gallery-dl][debug] Configuration Files ['C:\\Apps\\Tools - Utility\\DL-Files\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/feedfancier/'
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/feedfancier/'
[patreon][debug] TLS 1.2 disabled.
[patreon][debug] Loading cookies from 'C:/Apps/Tools - Utility/DL-Files/patreon-cookies.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /feedfancier/posts HTTP/11" 200 None
[patreon][debug] campaign_id: 104848
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=campaign,access_rules,attachments,audio,images,media,native_video_insights,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,user,user_defined_tags,ti_checks&fields%5Bcampaign%5D=currency,show_audio_post_download_links,avatar_photo_url,avatar_photo_image_urls,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Bpost%5D=change_visibility_at,comment_count,commenter_count,content,current_user_can_comment,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,insights_last_updated_at,is_paid,like_count,meta_image_url,min_cents_pledged_to_view,post_file,post_metadata,published_at,patreon_url,post_type,pledge_url,preview_asset_type,thumbnail,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner,has_ti_violation,moderation_status,post_level_suspension_removal_date,pls_one_liners_by_category,video_preview,view_count&fields%5Bpost_tag%5D=tag_type,value&fields%5Buser%5D=image_url,full_name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&fields%5Bmedia%5D=id,image_urls,download_url,metadata,file_name&fields%5Bnative_video_insights%5D=average_view_duration,average_view_pct,has_preview,id,last_updated_at,num_views,preview_views,video_duration&filter%5Bcampaign_id%5D=104848&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bis_draft%5D=false&sort=-published_at&json-api-version=1.0 HTTP/11" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/243223 HTTP/11" 200 None
[patreon][debug] Using download archive 'C:/Apps/Tools - Utility/DL-Files/archive/patreon_archive.sqlite3'
[patreon][debug] Active postprocessor modules: [MetadataPP]
[downloader.ytdl][debug] [generic] Extracting URL: https://stream.mux.com/WGmjYvJkDC67oHZ1cUNI5Cq5xctB6a93AQGu5awxPws.m3u8?token=eyJhbGciOiJSUzI1NiI...AbGrsSUCAfRgnVo4jXKw
[downloader.ytdl][debug] [generic] WGmjYvJkDC67oHZ1cUNI5Cq5xctB6a93AQGu5awxPws.m3u8?token=eyJhbGciOiJSUzI1NiIsImtpZCI6Ik5CY3o3Sk5RcUNmdDdWcmo5MWhra2lEY3Vyc2xtRGNmSU1oSFUzallZMDI0IiwidHlwIjoiSldUIn0.eyJzdWIiOiJXR21qWXZKa0RDNjdvSFoxY1VOSTVDcTV4Y3RCNmE5M0FRR3U1YXd4UHdzIiwiZXhwIjoxNzIyMTAzMjAwLCJhdWQiOiJ2IiwicGxheWJhY2tfcmVzdHJpY3Rpb25faWQiOiJJcjAyRm1xc3FVbk1JcEVxSDg5TWRsdldhMTVRd09vNmRDbmVsQ3RTSTlZSSJ9: Downloading webpage
[downloader.ytdl][error] ERROR: _parse_browser_specification() takes from 1 to 4 positional arguments but 7 were given
[download][error] Failed to download Feedfancier - 2024-07-24 14-41-30 (1), 'Star Fox Wolfs O'Donnell - 4K Animation' - Patreon.mp4

I similarly checked my yt-dlp config file (yt-dlp.txt), but even when I deleted everything it still output this error. I looked for anything that might be _parse_browser_specification() in the gdl config file, but I am not sure what that's in reference to, so I don't know why seven arguments are passed instead of four or less.

Thank you for helping me. I am sorry this is turning out to be such a puzzle.

mikf commented 3 months ago

[downloader.ytdl][error] ERROR: _parse_browser_specification() takes from 1 to 4 positional arguments but 7 were given

That's the --cookies-from-browser bug I mentioned in https://github.com/mikf/gallery-dl/issues/5885#issuecomment-2250872175. This will be fixed with the next git push.

As a workaround, remove / comment out the --cookies-from-browser line in your yt-dlp config.

ghost commented 3 months ago

@mikf That worked, thank you so much.

mikf commented 3 months ago

The --cookies-from-browser bug is fixed in https://github.com/gdl-org/builds/releases/tag/2024.07.26, so you should be able to re enable it (https://github.com/mikf/gallery-dl/commit/f321272b7c158181b252955e28d5c4c7e367c699).