Totonyus / ydl_api_ng

GNU General Public License v3.0
134 stars 15 forks source link

Succesful API request fails to trigger download #14

Closed rstrblstr closed 7 months ago

rstrblstr commented 7 months ago

I was super excited to find this, but keep running into trouble getting it to work. I rolled everything back to a default params.ini and still have the same issue.

using curl (and/or the userscript) successfully reaches the api, but nothing actually gets downloaded. I'm not seeing any errors that I recognize. Logs are set to debug.

[admin@server]# curl 'http://10.0.0.10:5011/download?url=https://www.youtube.com/watch?v=9Lgc3TxqgHA'

{"status_code":200,"url":"https://www.youtube.com/watch?v=9Lgc3TxqgHA","url_hostname":"www.youtube.com","no_preset_found":false,"presets_found":0,"presets_not_found":0,"all_downloads_checked":true,"passed_checks":1,"failed_checks":0,"downloads_can_be_checked":1,"downloads_cannot_be_checked":0,"ignore_post_security":false,"relaunch_failed_mode":null,"downloads":[{"_name":"DEFAULT","quiet":false,"noplaylist":true,"updatetime":false,"subtitleslangs":["en"],"subtitlesformat":"vtt","format":"bestvideo+bestaudio/best","restrictfilenames":false,"windowsfilenames":false,"no_color":true,"ignoreerrors":true,"_when_playlist":{"ignoreerrors":true},"paths":{"home":"./downloads/"},"outtmpl":{"default":"videos/%(webpage_url_domain)s/%(title)s_(%(height)s).%(ext)s"},"_hosts":["www.youtube.com","youtu.be"],"_video_indicators":["/watch?"],"_playlist_indicators":["?list=","&list=","/user/","/playlists","/videos"],"_default":true,"__check_exception_message":null,"__can_be_checked":true,"__check_result":true,"__is_video":true,"__is_playlist":false,"_redis_id":"998fa2d7-b10b-47ac-a09c-87b05664b23a"}],"programmation":null}

api.log shows the api receiving and starting the download, along with a 200 response

[30-12-23 00:12:30][download_manager][INFO] Init download - user_token: None - presets: None - url :https://www.youtube.com/watch?v=9Lgc3TxqgHA 
[30-12-23 00:12:30][config_manager][DEBUG] Merging preset DEFAULT in user YOUTUBE
[30-12-23 00:12:30][youtube-dlp][DEBUG] [youtube] Extracting URL: https://www.youtube.com/watch?v=9Lgc3TxqgHA
[30-12-23 00:12:30][youtube-dlp][DEBUG] [youtube] 9Lgc3TxqgHA: Downloading webpage
[30-12-23 00:12:31][youtube-dlp][DEBUG] [youtube] 9Lgc3TxqgHA: Downloading ios player API JSON
[30-12-23 00:12:31][youtube-dlp][DEBUG] [youtube] 9Lgc3TxqgHA: Downloading android player API JSON
[30-12-23 00:12:31][youtube-dlp][DEBUG] [youtube] 9Lgc3TxqgHA: Downloading m3u8 information
[30-12-23 00:12:31][youtube-dlp][DEBUG] [info] 9Lgc3TxqgHA: Downloading 1 format(s): 136+251
[30-12-23 00:12:31][download_manager][INFO] Checking url with preset DEFAULT => check passed
[30-12-23 00:12:31][pre_download_hooks][INFO] Downloading https://www.youtube.com/watch?v=9Lgc3TxqgHA with preset DEFAULT in {'default': 'videos/%(webpage_url_domain)s/%(title)s_(%(height)s).%(ext)s'}
[30-12-23 00:12:31][rq.queue][DEBUG] Pushed job 998fa2d7-b10b-47ac-a09c-87b05664b23a into ydl_api_ng
[30-12-23 00:12:31][uvicorn.access][INFO] 172.29.96.1:44256 - "GET /download?url=https://www.youtube.com/watch?v=9Lgc3TxqgHA HTTP/1.1" 200

yt-dlp direct from the command line works fine within the docker container

root@c4a76e827c25:/app# yt-dlp -f 'bestvideo[height<=1080][ext=mp4][fps<30][height<=1080][vcodec!~='vp']+bestaudio[ext=m4a]/best[height<=1080][ext=mp4][fps<30][height<=1080][vcodec!~='vp']/best' --embed-subs --embed-thumbnail --embed-metadata --write-info-json --write-playlist-metafiles --compat-options no-live-chat -o '/app/downloads/videos/%(channel)s [youtube-%(channel_id)s]/%(channel)s/%(title)s [youtube-%(id)s].%(ext)s' https://www.youtube.com/watch?v=9Lgc3TxqgHA 
[youtube] Extracting URL: https://www.youtube.com/watch?v=9Lgc3TxqgHA
[youtube] 9Lgc3TxqgHA: Downloading webpage
[youtube] 9Lgc3TxqgHA: Downloading ios player API JSON
[youtube] 9Lgc3TxqgHA: Downloading android player API JSON
[youtube] 9Lgc3TxqgHA: Downloading m3u8 information
[info] 9Lgc3TxqgHA: Downloading 1 format(s): 136+140
[info] There are no subtitles for the requested languages
[info] Downloading video thumbnail 41 ...
[info] Writing video thumbnail 41 to: /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].webp
[info] Writing video metadata as JSON to: /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].info.json
[download] Destination: /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].f136.mp4
[download] 100% of  276.44KiB in 00:00:00 at 432.49KiB/s
[download] Destination: /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].f140.m4a
[download] 100% of   44.40KiB in 00:00:00 at 113.66KiB/s
[Merger] Merging formats into "/app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].mp4"
Deleting original file /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].f136.mp4 (pass -k to keep)
Deleting original file /app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].f140.m4a (pass -k to keep)
[EmbedSubtitle] There aren't any subtitles to embed
[Metadata] Adding metadata to "/app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].mp4"
[ThumbnailsConvertor] Converting thumbnail "/app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].webp" to png
[EmbedThumbnail] mutagen: Adding thumbnail to "/app/downloads/videos/Totonyus [youtube-UCQexII-PYa3nOUYElK0sOPw]/Totonyus/Fichtre [youtube-9Lgc3TxqgHA].mp4"
root@c4a76e827c25:/app# 

my params.ini is default sample, except for disabling user management, changing the host, and switching to DEBUG logs

;;;;;
;;; - All parameters starting with an underscore (_) are technical parameters for this application
;;; - All others parameters are youtube-dl python parameters : https://github.com/yt-dlp/yt-dlp/blob/cda1bc51973c89b72b916dcc40dbe3d7f457097d/yt_dlp/YoutubeDL.py#L183
;;; - Sections type :
;;;     - [location:<NAME>]
;;;     - [template:<NAME>]
;;;     - [USER:<NAME>]
;;;     - [SITE:<NAME>]
;;;     - [AUTH:<NAME>]
;;;     - [PRESET:<NAME>]
;;; - Expansion tags :
;;;     - _location
;;;     - _template
;;;     - _user
;;;     - _site
;;;     - _auth
;;;     - _preset
;;; - Expand a section permit to merge the target section in the current section
;;; - If a parameter exists in current section, it will not be overriden
;;;
;;; - Precedence from lower to higher :
;;;   - preset
;;;   - user
;;;   - site
;;;
;;; - Every parameter can have different type, check params_metadata.ini to add or change parameters type
;;;
;;;;;

[app]
_api_route_download = /download
_api_route_queue = /queue
_api_route_extract_info = /extract_info
_api_route_info = /info
_api_route_active_downloads = /active_downloads
_enable_users_management = false
;; 0 : NOTSET, 10 : DEBUG, 20 : INFO; 30 : WARNING; 40 : ERROR; 50 : CRITICAL
_log_level = 10
;; number of log files to keep
_log_backups = 7
_listen_port = 80
_listen_ip = 0.0.0.0
;; in seconds, time between each time the programmation feature will check what to launch
_programmation_interval = 60
_enable_redis = true
_redis_ttl = 3600
_redis_host = 10.0.0.10
_redis_port = 6379

;;; just for unit tests purpose, please don't mind
_int_test = 42
_float_test = 42.0
_bool_test = true
_string_test = test
_array_test = 4,2
_object_test = {"nb": "42"}

;; If false, hosts and outtmpl will be ignored in post requests and replaced by default configuration
_allow_dangerous_post_requests = false

;;;
;;; Locations
;;; Only contains the path
;;; Must keep a DEFAULT location
;;; Docker note : you must keep the _path in /app/ directory
;;; If you are not using docker, you can use whatever you want
;;;
[location:DEFAULT]
paths = {"home" : "./downloads/"}

[location:DAD]
paths = {"home" : "./downloads/symlink_to_dad_home/"}

[location:SHARE]
paths = {"home" : "./downloads/symlink_to_some_shared_folder/or_a_nas/or_something/ya_know/"}

;;;
;;; Templates
;;; Contains the output template of the filename
;;; Must keep a DEFAULT template
;;;
[template:DEFAULT]
outtmpl = {"default" : "videos/%(webpage_url_domain)s/%(title)s_(%(height)s).%(ext)s"}

[template:AUDIO]
outtmpl = {"default" : "audio/%(title)s.%(ext)s", "chapter" : "audio/%(title)s/%(section_number)s-%(section_title)s.%(ext)s"}

[template:PLAYLIST]
outtmpl = {"default" : "videos/%(webpage_url_domain)s/playlists/%(playlist)s/%(title)s_(%(height)s).%(ext)s"}

;;;
;;; Presets
;;; Must keep a DEFAULT preset
;;; Every other preset will automatically be extended with the DEFAULT preset
;;;
[preset:DEFAULT]
; Will use the DEFAULT location item
_location = DEFAULT
; Will use the DEFAULT template item
_template = DEFAULT

; youtube-dlp parameters :
quiet = false
noplaylist = true
updatetime = false
subtitleslangs = en
subtitlesformat = vtt
format = bestvideo+bestaudio/best
restrictfilenames = false
windowsfilenames = false
no_color = true
ignoreerrors = false
;;; options to add only when downloading a playlist
;;; here to stop breaking errors handling on regular videos without breaking playlists download
_when_playlist = {"ignoreerrors": true}

;;; Only need to override parameters that changed from preset
[preset:AUDIO]
; will use AUDIO template to have a different naming convention
_template = AUDIO
format = bestaudio
; convert to mp3 then split each track when the video contains chapters
postprocessors = [{"key": "FFmpegExtractAudio", "preferredcodec": "mp3", "preferredquality": "320"}, {"key": "EmbedThumbnail"},{"key" : "FFmpegMetadata"}, {"key": "FFmpegSplitChapters", "force_keyframes": false}]
writethumbnail = true

[preset:BEST]
format = bestvideo+bestaudio/best

[preset:FULLHD]
format = best[height=1080]/bestvideo[height=1080]+bestaudio/best

[preset:HD]
format = best[height=720]/bestvideo[height=720]+bestaudio/best

[preset:SD]
format = best[height=360]/bestvideo[height=360]+bestaudio/best

[preset:SHARE]
_location = SHARE
; you can extend another preset
_preset = HD

[preset:PLAYLIST]
_template = PLAYLIST
_preset = HD
noplaylist = false

;;;
;;; Users
;;;
[user:DAD]
_location = DAD
; The token is mandatory to find the right user
; Not two users can have the same token
_token = dad_super_password

[user:MOM]
; Files will be downloaded in Dad location
_location = DAD
_token = mom_super_password
; Mom like to have her subtitles
writesubtitles = true

[user:NONE]
_location = DEFAULT
_token = XXXXXXXXX
_allow_programmation = true
subtitleslangs = en
writesubtitles = true

;;;
;;; Sites
;;; Used to dertermine a few special tricks for websites
;;; If exanding an auth section, auth section will not override site parameters
;;; Any parameter in this section will always override existing parameters
;;;
[site:KNOWN]
ignoreerrors = true

[site:YOUTUBE]
_hosts = www.youtube.com,youtu.be
_video_indicators = /watch?
;; download check will not be performed on playlists (youtube-dl checks every video of the playlist)
_playlist_indicators = ?list=,&list=,/user/,/playlists,/videos
;; You can add parameters for site only, here, I want to get thumbnails only on youtube videos
; writethumbnail = true
;; As we know how to detect if it's a playlist or not, we can set ignoreerrors to false to have a better errors handling
;; As _when_playlist is defined in DEFAULT preset, it will be override in case of playlist
_site : KNOWN

[site:DAILYMOTION]
_host = www.dailymotion.com,dailymotion.com
_auth = DAILYMOTION

;;;
;;; Auth
;;; Every parameter in auth section will override existing parameters
;;; Meant to the authentication layer
;;;
[auth:DAILYMOTION]
username = Totonyus
password = is_this_site_still_running_fr_?
rstrblstr commented 7 months ago

I rolled everything back to the most basic components of params.ini.

I'm not sure what was causing the problem, but it works now with this:

[app]
_api_route_download = /download
_api_route_queue = /queue
_api_route_extract_info = /extract_info
_api_route_info = /info
_api_route_active_downloads = /active_downloads
_enable_users_management = false
_log_level = 10
_log_backups = 7
_listen_port = 80
_listen_ip = 0.0.0.0
_enable_redis = true
_redis_ttl = 3600
_redis_host = ydl_api_ng_redis
_redis_port = 6379

_allow_dangerous_post_requests = false

;;;
;;; Locations
;;;
[location:DEFAULT]
paths = {"home" : "./downloads/"}

;;;
;;; Templates
;;;
[template:DEFAULT]
outtmpl = {"default" : "videos/%(channel)s_[youtube-%(channel_id)s]/%(channel)s/%(title)s_[youtube-%(id)s].%(ext)s"}

[template:AUDIO]
outtmpl = {"default" : "audio/%(title)s.%(ext)s", "chapter" : "audio/%(title)s/%(section_number)s-%(section_title)s.%(ext)s"}

[template:PLAYLIST]
outtmpl = {"default" : "videos/%(channel)s_[youtube-%(channel_id)s]/%(playlist)s_[youtube-%(playlist-id)s]/%(title)s_[youtube-%(id)s].%(ext)s"}

;;;
;;; Presets
;;;
[preset:DEFAULT]
_location = DEFAULT
_template = DEFAULT

; youtube-dlp default parameters :
quiet = true
noplaylist = true
updatetime = false
subtitleslangs = en
format = bestvideo[height<=1080][ext=mp4][fps<30][vcodec!~='vp']+bestaudio[ext=m4a]/best[ext=mp4][fps<30][height<=1080][vcodec!~='vp']/best
restrictfilenames = true
windowsfilenames = true
ignoreerrors = true
writeinfojson = true
writesubtitles = true
writethumbnail = true
postprocessors = [{"key": "SponsorBlock"}, {"key" : "FFmpegEmbedSubtitle", "already_have_subtitle": true}, {"key": "ModifyChapters", "sponsorblock_chapter_title" : true, "force_keyframes" : true},  {"key" : "FFmpegMetadata"}, {"key": "FFmpegSplitChapters", "force_keyframes": false}, {"key": "EmbedThumbnail", "already_have_thumbnail": true}]
_when_playlist = {"ignoreerrors" : true}
cachedir = /home/ydl_api_ng/cache

[preset:AUDIO]
_template = AUDIO
format = bestaudio
postprocessors = [{"key": "FFmpegExtractAudio", "preferredcodec": "mp3", "preferredquality": "320"}, {"key": "EmbedThumbnail"}, {"key" : "FFmpegMetadata"}, {"key": "FFmpegSplitChapters", "force_keyframes": false}]
writethumbnail = true

[preset:BEST]
;;; format = bestvideo[height<=1080][ext=mp4][fps<30][vcodec!~='vp']+bestaudio[ext=m4a]/best[ext=mp4][fps<30][height<=1080][vcodec!~='vp']/best
_template = DEFAULT
_preset = DEFAULT

[preset:FULLHD]
;;; format = bestvideo[height<=1080][ext=mp4][fps<30][vcodec!~='vp']+bestaudio[ext=m4a]/best[ext=mp4][fps<30][height<=1080][vcodec!~='vp']/best
_template = DEFAULT
_preset = DEFAULT

[preset:HD]
format = bestvideo[height<=720][ext=mp4][fps<30][vcodec!~='vp']+bestaudio[ext=m4a]/best[ext=mp4][fps<30][height<=720][vcodec!~='vp']/best
_template = DEFAULT
_preset = DEFAULT

[preset:SD]
format = bestvideo[height<=480][ext=mp4][fps<30][vcodec!~='vp']+bestaudio[ext=m4a]/best[ext=mp4][fps<30][height<=480][vcodec!~='vp']/best
_template = DEFAULT
_preset = DEFAULT

[preset:PLAYLIST]
_template = PLAYLIST
_preset = BEST
noplaylist = false
allow_playlist_files = true

;;;
;;; Sites
;;;
[site:KNOWN]
ignoreerrors = false

[site:YOUTUBE]
_hosts = www.youtube.com,youtu.be
_video_indicators = /watch?
_playlist_indicators = ?list=,&list=,/user/,/playlists,/videos
_site = KNOWN
cookiefile = /app/cookies/www.youtube.com_cookies.txt