No srt file downloaded after switching to --HEAD to overcome the uploader id warning

simkimsia commented 1 year ago

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

kimsia@kimsia-MBP2021 ~ % youtube-dl --write-auto-sub --sub-format srt --skip-download 'https://www.youtube.com/watch?v=2Wn-OJSgVQs' --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--write-auto-sub', '--sub-format', 'srt', '--skip-download', 'https://www.youtube.com/watch?v=2Wn-OJSgVQs', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 763c41f00
[debug] Python version 3.11.3 (CPython) - macOS-13.3.1-arm64-arm-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] 2Wn-OJSgVQs: Downloading webpage
WARNING: unable to extract uploader id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Default format spec: best/bestvideo+bestaudio

Description

I know this falls under the uploader id which is already fixed https://github.com/ytdl-org/youtube-dl/issues/31530

see the original log from above.

I have also reinstalled using homebrew by first uninstalling then reinstalling youtube-dl

kimsia@kimsia-MBP2021 ~ % brew uninstall youtube-dl
Uninstalling /opt/homebrew/Cellar/youtube-dl/2021.12.17... (2,412 files, 22.7MB)
kimsia@kimsia-MBP2021 ~ % brew install youtube-dl
==> Fetching youtube-dl
==> Downloading https://ghcr.io/v2/homebrew/core/youtube-dl/manifests/2021.12.17
Already downloaded: /Users/kimsia/Library/Caches/Homebrew/downloads/af4e009121ff7543121df7c09854a632e93503182cf834b087ae0a5c76d6c62d--youtube-dl-2021.12.17-2.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/youtube-dl/blobs/sha256:1785d92
Already downloaded: /Users/kimsia/Library/Caches/Homebrew/downloads/ccb915ea24581bba9844d03ed11173ef62346d1d436c136e84466ccb2104df69--youtube-dl--2021.12.17.arm64_ventura.bottle.2.tar.gz
==> Pouring youtube-dl--2021.12.17.arm64_ventura.bottle.2.tar.gz
🍺  /opt/homebrew/Cellar/youtube-dl/2021.12.17: 1,596 files, 16.3MB
==> Running `brew cleanup youtube-dl`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

I have also read the instruction at the comment https://github.com/ytdl-org/youtube-dl/issues/31530#issuecomment-1520185289 which recommends that I tried nightly build.

So I did.

kimsia@kimsia-MBP2021 ~ % brew install youtube-dl --HEAD
==> Fetching youtube-dl
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/df26d4c
Already downloaded: /Users/kimsia/Library/Caches/Homebrew/downloads/a84e27d22577fbefdf8bd9a9cc680b8208f2525813c3e71cff110e628a1ac3c1--youtube-dl.rb
==> Cloning https://github.com/ytdl-org/youtube-dl.git
Updating /Users/kimsia/Library/Caches/Homebrew/youtube-dl--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 211cbfd5d [jsinterp] Minimally handle arithmetic operator precedence
==> make PREFIX=/opt/homebrew/Cellar/youtube-dl/HEAD-211cbfd MANDIR=/opt/homebre
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/youtube-dl/HEAD-211cbfd: 11 files, 2.3MB, built in 6 seconds
==> Running `brew cleanup youtube-dl`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /Users/kimsia/Library/Caches/Homebrew/youtube-dl--ruby-source--HEAD.rb... (2.6KB)
Removing: /Users/kimsia/Library/Caches/Homebrew/youtube-dl--2021.12.17... (4.3MB)

Now the error is overcome, but no visible output. As in I cannot see any files of the subtitles downloaded. I have searched my system using the 2Wn-OJSgVQs as part of filename

kimsia@kimsia-MBP2021 ~ % youtube-dl --write-auto-sub --sub-format srt --skip-download 'https://www.youtube.com/watch?v=2Wn-OJSgVQs' --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--write-auto-sub', '--sub-format', 'srt', '--skip-download', 'https://www.youtube.com/watch?v=2Wn-OJSgVQs', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.11.3 (CPython) - macOS-13.3.1-arm64-arm-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] 2Wn-OJSgVQs: Downloading webpage
[youtube] 2Wn-OJSgVQs: Downloading player d87d581f
[debug] [youtube] Decrypted nsig LHkD7B1qQHsMmYkLr => fq_ndnspIZdpeg
[debug] [youtube] Decrypted nsig ZSIO6EZeUcBThbV9q => lHBsNR4Bmvjuxg
[debug] Default format spec: best/bestvideo+bestaudio

Stil nothing. I am on macOS 13.3.1 (22E261)

dirkf commented 1 year ago

YT automatic captions may not be available: maybe the uploader disabled them, maybe YT hasn't finished creating them.

But this video has actual subtitles, not in SRT format, but --convert-subs works. However it's ignored when you say --skip-download (why? -- see below). Normally if you've already downloaded the target video with the same filename template, the idempotent operation of yt-dl should just skip the download anyway (still pointlessly decoding YT download links ...), so you don't need to skip it explicitly.

In the example below, as I hadn't downloaded the video, I used --test to download just a few kB: don't do that if you have a full download already.

$ python -m youtube_dl --list-subs 'https://www.youtube.com/watch?v=2Wn-OJSgVQs'
[youtube] Extracting URL: https://www.youtube.com/watch?v=2Wn-OJSgVQs...
[youtube] 2Wn-OJSgVQs: Downloading webpage
Available subtitles for 2Wn-OJSgVQs:
Language formats
en       vtt, ttml, srv3, srv2, srv1, json3
$ python -m youtube_dl --write-sub --sub-format vtt --convert-subs srt --test 'https://www.youtube.com/watch?v=2Wn-OJSgVQs' --verbose
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--write-sub', u'--sub-format', u'vtt', u'--convert-subs', u'srt', u'--test', u'https://www.youtube.com/watch?v=2Wn-OJSgVQs', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 64d6dd64c
[debug] Python version 2.7.18 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] Extracting URL: https://www.youtube.com/watch?v=2Wn-OJSgVQs
[youtube] 2Wn-OJSgVQs: Downloading webpage
[debug] [youtube] Decrypted nsig phwVRJap3We5e8wsW => kL58OEAmGTrawg
[debug] [youtube] Decrypted nsig UrbnAqDsQWpcDU8p7 => y0o4WdHM29m6FA
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.en.vtt
[debug] Invoking downloader on u'https://rr2---sn-cu-cgne.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=mhxJZLOiBbfK0wWts7C4Bg&ip=146.198.107.110&clen=1888412100&id=o-AIFTdOQQS5ZzKaa4xVeHGAFWUKn9vWU2EGhAHqpqkpel&txp=4432434&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRQIhAIh86mHrGoFlScyw4d_S0-oaSe7xoILr4XsC0G0VqXYdAiBLiGb_F_q-WSQzVOK6PJSmT2shGAj7flaOkWhZt2gVoQ%3D%3D&dur=1282.875&ns=QwSOZydfkB1y4w0YY-T2LMEN&initcwndbps=1432500&vprv=1&lsig=AG3C_xAwRAIgU4VfLie1ruKrbnNSMaOLicT7CL8GxxFjptrrtyRSASsCIBcWBz6sRr5e8glv7KaUMWCPt_sqzt2vxyBT1sHITcle&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1682258372217742&c=WEB&expire=1682534650&mime=video%2Fwebm&fvip=5&itag=313&mm=31%2C29&mn=sn-cu-cgne%2Csn-cu-c9i6&mh=uw&n=kL58OEAmGTrawg&mt=1682512627&mv=m&pl=25&ms=au%2Crdu&mvi=2'
[dashsegments] Total fragments: 1
[download] Destination: When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f313.webm
[download] 100% of 10.00KiB in 00:00
[debug] Invoking downloader on u'https://rr2---sn-cu-cgne.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&fexp=24007246&ei=mhxJZLOiBbfK0wWts7C4Bg&ip=146.198.107.110&clen=21449572&id=o-AIFTdOQQS5ZzKaa4xVeHGAFWUKn9vWU2EGhAHqpqkpel&txp=4432434&gir=yes&requiressl=yes&keepalive=yes&source=youtube&sig=AOq0QJ8wRgIhAJ3QrhuETolvHVVf_6I0Fl3Yq0FrUuXL2Tm4JTK9wd33AiEArg7JirHwwbLUiRPQhP73PAgeuaC4j7Ajbntn4OCv6pE%3D&dur=1282.901&ns=QwSOZydfkB1y4w0YY-T2LMEN&initcwndbps=1432500&vprv=1&lsig=AG3C_xAwRAIgU4VfLie1ruKrbnNSMaOLicT7CL8GxxFjptrrtyRSASsCIBcWBz6sRr5e8glv7KaUMWCPt_sqzt2vxyBT1sHITcle&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1682242171138419&c=WEB&expire=1682534650&mime=audio%2Fwebm&fvip=5&itag=251&mm=31%2C29&mn=sn-cu-cgne%2Csn-cu-c9i6&mh=uw&n=kL58OEAmGTrawg&mt=1682512627&mv=m&pl=25&ms=au%2Crdu&mvi=2'
[dashsegments] Total fragments: 1
[download] Destination: When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f251.webm
[download] 100% of 10.00KiB in 00:00
[ffmpeg] Merging formats into "When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f313.webm' -i 'file:When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f251.webm' -c copy -map '0:v:0' -map '1:a:0' 'file:When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.temp.webm'
Deleting original file When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f313.webm (pass -k to keep)
Deleting original file When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.f251.webm (pass -k to keep)
[ffmpeg] Converting subtitles
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.en.vtt' -f srt 'file:When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.en.srt'
Deleting original file When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.en.vtt (pass -k to keep)
$

dirkf commented 1 year ago

... --convert-subs [is] ignored when you say -skip-download (why?). ...

Because no postprocessing is performed when -skip-download is specified.

This appears to work the same in yt-dlp.

pukkandan commented 1 year ago

This appears to work the same in yt-dlp.

No, this was changed in 57df9f53e0159e99ff8ff55d9f761291b8aaecce, 56d868dbb7c72e4fbe9d28d4837cc59261d8fe55

❯ yt-dlp --write-auto-sub --convert-sub srt --skip-download "https://www.youtube.com/watch?v=2Wn-OJSgVQs" --verbose
[debug] Command-line config: ['--ignore-config', '--write-auto-sub', '--convert-sub', 'srt', '--skip-download', 'https://www.youtube.com/watch?v=2Wn-OJSgVQs', '--verbose']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 7cf51f219
[debug] Python 3.11.0 (CPython AMD64 64bit) - Windows-10-10.0.22000-SP0 (OpenSSL 1.1.1q  5 Jul 2022)
[debug] exe versions: ffmpeg n5.1.2-8-g5746987bad-20221204 (fdk,setts), ffprobe N-109332-g45ab5307a6-20221201, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1810 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=2Wn-OJSgVQs
[youtube] 2Wn-OJSgVQs: Downloading webpage
[youtube] 2Wn-OJSgVQs: Downloading android player API JSON
[debug] Loading youtube-nsig.73d31b49 from cache
[debug] [youtube] Decrypted nsig 2odPxdR8y9bVP-fnlqO => rRO7WJlpO7h61Q
[info] 2Wn-OJSgVQs: Downloading subtitles: en-en
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 2Wn-OJSgVQs: Downloading 1 format(s): 313+251
Deleting existing file When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.vtt
[info] Writing video subtitles to: When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.vtt
[debug] Invoking http downloader on "https://www.youtube.com/api/timedtext?v=2Wn-OJSgVQs&caps=asr&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1682588670&sparams=ip%2Cipbits%2Cexpire%2Cv%2Ccaps%2Cxoaf&signature=CEEB7695F3CC7BD3A814B1A5163468D727DD4035.AFE9F73579DD67EEA5975A0C6140F2EEAD19BA1D&key=yt8&lang=en&fmt=vtt"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.vtt
[download] 100% of   21.95KiB in 00:00:00 at 113.13KiB/s
[SubtitlesConvertor] Converting subtitles
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.vtt" -f srt -movflags "+faststart" "file:When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.srt"
Deleting original file When Two Filmmakers Make the Same Movie [2Wn-OJSgVQs].en-en.vtt (pass -k to keep)

dirkf commented 1 year ago

OK, thanks, because yt-dlp runs PPs outside the code controlled by skip_download via (in this case)

        new_info, files_to_move = self.pre_process(info_dict, 'before_dl', files_to_move)

simkimsia commented 1 year ago

so what you're saying is @dirkf is that

--convert-subs is ignored when you say --skip-download
i should have done a --list-subs first
then do a convert-subs but with the video downloaded as well

Am I right?

What I tried

the list-subs

youtube-dl --list-subs 'https://www.youtube.com/watch?v=2Wn-OJSgVQs'
[youtube] 2Wn-OJSgVQs: Downloading webpage
[youtube] 2Wn-OJSgVQs: Downloading player 73d31b49
Available subtitles for 2Wn-OJSgVQs:
Language formats
en       vtt, ttml, srv3, srv2, srv1, json3

the convert-subs with the video downloaded

kimsia@kimsia-MBP2021 ~ % youtube-dl --write-sub --sub-format vtt --convert-subs srt  'https://www.youtube.com/watch?v=2Wn-OJSgVQs' --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--write-sub', '--sub-format', 'vtt', '--convert-subs', 'srt', 'https://www.youtube.com/watch?v=2Wn-OJSgVQs', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.11.3 (CPython) - macOS-13.3.1-arm64-arm-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] 2Wn-OJSgVQs: Downloading webpage
[debug] [youtube] Decrypted nsig 6BxDCzrm2cpS-Uk2-8Y => ev_TEzI4Ykld2w
[debug] [youtube] Decrypted nsig 7zKqt0jJTVWZ0sGjJRh => rNZ0l6llh-mMLA
[debug] Default format spec: best/bestvideo+bestaudio
[info] Writing video subtitles to: When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.en.vtt
[debug] Invoking downloader on 'https://rr5---sn-5fo-c33ee.googlevideo.com/videoplayback?expire=1682590290&ei=8vVJZNdWyaLPuw_3ra_4DA&ip=184.22.202.193&id=o-AHQ7BSx9CfGgRT7PjxnsEBQPnX7b7ACqmeGfOhswOeGW&itag=22&source=youtube&requiressl=yes&mh=uw&mm=31%2C26&mn=sn-5fo-c33ee%2Csn-npoldn76&ms=au%2Conr&mv=m&mvi=5&pl=22&pcm2=yes&initcwndbps=2692500&vprv=1&mime=video%2Fmp4&ns=ZBz2WZFF7Exsfurnn9NdMLcN&cnr=14&ratebypass=yes&dur=1282.925&lmt=1682549048669153&mt=1682568306&fvip=2&fexp=24007246&beids=24512784&c=WEB&txp=5532434&n=rNZ0l6llh-mMLA&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cpcm2%2Cvprv%2Cmime%2Cns%2Ccnr%2Cratebypass%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAL7s4w1SvEETTMhSm1JU6-kYihWeWaxTBwVSyErUhZxbAiAb4gBZQI2dGN0KB3lGyLmriX2JAS5-EKnfc5q2X94OcQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAKQ7VlSveicqYOGgzPEK9Yp9U9VCBV9Jwuy5lJRLEWfxAiAag97zJSWCP76Cy1Dg3Mn91Yf7GMsOE3l_bvEKhUwirA%3D%3D'
[download] Destination: When Two Filmmakers Make the Same Movie-2Wn-OJSgVQs.mp4
[download] 100% of 144.08MiB in 00:27
[ffmpeg] Converting subtitles

This works.

I am closing this.

dirkf commented 1 year ago

--list-subs is just to show what subtitles are available. From the logs posted here, I think many subtitle enthusiasts just use --write-sub --write-auto-sub and take what they get.
--convert-subs is disabled by --skip-download, so don't use both together. If you want --convert-subs, you have to expect the video to be downloaded as well, except that yt-dl won't do that if you already have the file downloaded with the same file name template (or a similar one, such that the downloaded file would have the same name as the existing one). Generally, people download a video with its converted subtitles in one operation, so this problem doesn't arise.
A possible enhancement, implemented twice in major forks (as noted above), would be to stop --skip-download from disabling --convert-subs, but this is difficult if people have scripts that depend on the current behaviour.

ytdl-org / youtube-dl