2 videos "failed to download: 'NoneType' object is not subscriptable" — yt-dlp `--format best` fails to download complete video (same 1503232-byte .mp4.part fragment every time!)

holta commented 8 months ago

Reposting error from #134:

failed to download: 'NoneType' object is not subscriptable

Videos are:

@deldesir mentioned:

Generally when the video is playable and Type error is returned, I assume the requested format is not found.

114

holta commented 8 months ago

This appears to be a yt-dlp bug with --format best — regardless whether or not --format-sort 'tbr~1000' is also used — here's the evidence:

root@box:~# yt-dlp --format best --format-sort 'tbr~1000' https://youtu.be/ZfbiLfSczgA
WARNING: "-f best" selects the best pre-merged format which is often not the best option.
         To let yt-dlp download and merge the best available formats, simply do not pass any format selection.
         If you know what you are doing and want only the best pre-merged format, use "-f b" instead to suppress this warning
[youtube] Extracting URL: https://youtu.be/ZfbiLfSczgA
[youtube] ZfbiLfSczgA: Downloading webpage
[youtube] ZfbiLfSczgA: Downloading ios player API JSON
[youtube] ZfbiLfSczgA: Downloading android player API JSON
[youtube] ZfbiLfSczgA: Downloading m3u8 information
[info] ZfbiLfSczgA: Downloading 1 format(s): 22
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.5% of  184.18MiB at   10.80MiB/s ETA 00:16[download] Got error: 1529438 bytes read, 8702693 more expected. Retrying (1/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.7% of  184.18MiB at   11.94MiB/s ETA 00:15[download] Got error: 481886 bytes read, 9529031 more expected. Retrying (2/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.7% of  184.18MiB at    6.90MiB/s ETA 00:26[download] Got error: 179893 bytes read, 9975172 more expected. Retrying (3/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.8% of  184.18MiB at    4.81MiB/s ETA 00:38[download] Got error: 90718 bytes read, 10235576 more expected. Retrying (4/10)...
[download] Got error: 0 bytes read, 10236524 more expected. Retrying (5/10)...
[download] Got error: 0 bytes read, 10255890 more expected. Retrying (6/10)...
[download] Got error: 0 bytes read, 10103802 more expected. Retrying (7/10)...
[download] Got error: 0 bytes read, 10060251 more expected. Retrying (8/10)...
[download] Got error: 0 bytes read, 10016190 more expected. Retrying (9/10)...
[download] Got error: 0 bytes read, 10126505 more expected. Retrying (10/10)...
[download] Got error: 0 bytes read, 10144969 more expected. Giving up after 10 retries

root@box:~# ls -l 039*
-rw-r--r-- 1 root root 1503232 Feb 22 13:38 '039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4.part'

Exact same result with just --format best here:

root@box:~# rm 039*

root@box:~# yt-dlp --format best https://youtu.be/ZfbiLfSczgA
WARNING: "-f best" selects the best pre-merged format which is often not the best option.
         To let yt-dlp download and merge the best available formats, simply do not pass any format selection.
         If you know what you are doing and want only the best pre-merged format, use "-f b" instead to suppress this warning
[youtube] Extracting URL: https://youtu.be/ZfbiLfSczgA
[youtube] ZfbiLfSczgA: Downloading webpage
[youtube] ZfbiLfSczgA: Downloading ios player API JSON
[youtube] ZfbiLfSczgA: Downloading android player API JSON
[youtube] ZfbiLfSczgA: Downloading m3u8 information
[info] ZfbiLfSczgA: Downloading 1 format(s): 22
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.5% of  184.18MiB at   22.03MiB/s ETA 00:08[download] Got error: 1529438 bytes read, 8750089 more expected. Retrying (1/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.7% of  184.18MiB at    6.90MiB/s ETA 00:26[download] Got error: 457397 bytes read, 9692810 more expected. Retrying (2/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.7% of  184.18MiB at    4.28MiB/s ETA 00:42[download] Got error: 179893 bytes read, 9893025 more expected. Retrying (3/10)...
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4
[download]   0.8% of  184.18MiB at    3.25MiB/s ETA 00:56[download] Got error: 90718 bytes read, 10392246 more expected. Retrying (4/10)...
[download] Got error: 0 bytes read, 10229572 more expected. Retrying (5/10)...
[download] Got error: 0 bytes read, 10354219 more expected. Retrying (6/10)...
[download] Got error: 0 bytes read, 10089105 more expected. Retrying (7/10)...
[download] Got error: 0 bytes read, 10356388 more expected. Retrying (8/10)...
[download] Got error: 0 bytes read, 10340869 more expected. Retrying (9/10)...
[download] Got error: 0 bytes read, 10316345 more expected. Retrying (10/10)...
[download] Got error: 0 bytes read, 9989908 more expected. Giving up after 10 retries

root@box:~# ls -l 039*
-rw-r--r-- 1 root root 1503232 Feb 22 10:37 '039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].mp4.part'

Conversely when removing the --format best parameter, a giant 158MB .webm file is downloaded:

root@box:~# yt-dlp https://youtu.be/ZfbiLfSczgA
[youtube] Extracting URL: https://youtu.be/ZfbiLfSczgA
[youtube] ZfbiLfSczgA: Downloading webpage
[youtube] ZfbiLfSczgA: Downloading ios player API JSON
[youtube] ZfbiLfSczgA: Downloading android player API JSON
[youtube] ZfbiLfSczgA: Downloading m3u8 information
[info] ZfbiLfSczgA: Downloading 1 format(s): 247+251
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].f247.webm
[download] 100% of   98.46MiB in 00:00:09 at 9.96MiB/s
[download] Destination: 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].f251.webm
[download] 100% of   58.61MiB in 00:00:05 at 10.32MiB/s
[Merger] Merging formats into "039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].webm"
Deleting original file 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].f247.webm (pass -k to keep)
Deleting original file 039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].f251.webm (pass -k to keep)

root@box:~# ls -lh 039*
-rw-r--r-- 1 root root 158M Jan  7 02:36 '039 – ALEVEL APPLIED MATHEMATICS｜ SIMPLE HARMONIC MOTION (MECHANICS)｜ FOR SENIOR 5 & 6 [ZfbiLfSczgA].webm'

Finally, the list of available / downloadable formats would appear to be completely normal:

root@box:~# yt-dlp -F https://youtu.be/ZfbiLfSczgA
[youtube] Extracting URL: https://youtu.be/ZfbiLfSczgA
[youtube] ZfbiLfSczgA: Downloading webpage
[youtube] ZfbiLfSczgA: Downloading ios player API JSON
[youtube] ZfbiLfSczgA: Downloading android player API JSON
[youtube] ZfbiLfSczgA: Downloading m3u8 information
[info] Available formats for ZfbiLfSczgA:
ID  EXT   RESOLUTION FPS CH │   FILESIZE   TBR PROTO │ VCODEC          VBR ACODEC      ABR ASR MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
sb3 mhtml 48x27        0    │                  mhtml │ images                                  storyboard
sb2 mhtml 80x45        0    │                  mhtml │ images                                  storyboard
sb1 mhtml 160x90       0    │                  mhtml │ images                                  storyboard
sb0 mhtml 320x180      0    │                  mhtml │ images                                  storyboard
233 mp4   audio only        │                  m3u8  │ audio only          unknown             [en] Default
234 mp4   audio only        │                  m3u8  │ audio only          unknown             [en] Default
599 m4a   audio only      2 │   15.89MiB   31k https │ audio only          mp4a.40.5   31k 22k [en] ultralow, m4a_dash
600 webm  audio only      2 │   18.04MiB   35k https │ audio only          opus        35k 48k [en] ultralow, webm_dash
139 m4a   audio only      2 │   25.18MiB   49k https │ audio only          mp4a.40.5   49k 22k [en] low, m4a_dash
249 webm  audio only      2 │   26.23MiB   51k https │ audio only          opus        51k 48k [en] low, webm_dash
250 webm  audio only      2 │   34.31MiB   66k https │ audio only          opus        66k 48k [en] low, webm_dash
140 m4a   audio only      2 │   66.84MiB  129k https │ audio only          mp4a.40.2  129k 44k [en] medium, m4a_dash
251 webm  audio only      2 │   58.61MiB  114k https │ audio only          opus       114k 48k [en] medium, webm_dash
269 mp4   256x144     15    │ ~ 99.34MiB  188k m3u8  │ avc1.4D400B    188k video only
597 mp4   256x144     15    │    9.33MiB   18k https │ avc1.4d400b     18k video only          144p, mp4_dash
160 mp4   256x144     15    │   12.03MiB   23k https │ avc1.4D400B     23k video only          144p, mp4_dash
602 mp4   256x144     15    │ ~ 66.59MiB  126k m3u8  │ vp09.00.10.08  126k video only
603 mp4   256x144     15    │ ~ 96.80MiB  183k m3u8  │ vp09.00.11.08  183k video only
598 webm  256x144     15    │   11.18MiB   22k https │ vp9             22k video only          144p, webm_dash
278 webm  256x144     15    │   13.59MiB   26k https │ vp09.00.11.08   26k video only          144p, webm_dash
229 mp4   426x240     15    │ ~172.49MiB  326k m3u8  │ avc1.4D4015    326k video only
133 mp4   426x240     15    │   25.22MiB   49k https │ avc1.4D4015     49k video only          240p, mp4_dash
604 mp4   426x240     15    │ ~143.25MiB  271k m3u8  │ vp09.00.20.08  271k video only
242 webm  426x240     15    │   21.06MiB   41k https │ vp09.00.20.08   41k video only          240p, webm_dash
230 mp4   640x360     15    │ ~394.27MiB  746k m3u8  │ avc1.4D4016    746k video only
134 mp4   640x360     15    │   43.50MiB   84k https │ avc1.4D4016     84k video only          360p, mp4_dash
18  mp4   640x360     15  2 │   93.55MiB  181k https │ avc1.42001E         mp4a.40.2       44k [en] 360p
605 mp4   640x360     15    │ ~248.62MiB  470k m3u8  │ vp09.00.21.08  470k video only
243 webm  640x360     15    │   40.48MiB   78k https │ vp09.00.21.08   78k video only          360p, webm_dash
231 mp4   854x480     15    │ ~706.12MiB 1336k m3u8  │ avc1.4D401E   1336k video only
135 mp4   854x480     15    │   71.20MiB  138k https │ avc1.4D401E    138k video only          480p, mp4_dash
606 mp4   854x480     15    │ ~367.89MiB  696k m3u8  │ vp09.00.30.08  696k video only
244 webm  854x480     15    │   60.21MiB  117k https │ vp09.00.30.08  117k video only          480p, webm_dash
22  mp4   1280x720    15  2 │ ≈188.60MiB  357k https │ avc1.64001F         mp4a.40.2       44k [en] 720p
232 mp4   1280x720    15    │ ~  1.27GiB 2464k m3u8  │ avc1.64001F   2464k video only
136 mp4   1280x720    15    │  117.63MiB  228k https │ avc1.64001F    228k video only          720p, mp4_dash
609 mp4   1280x720    15    │ ~577.90MiB 1093k m3u8  │ vp09.00.31.08 1093k video only
247 webm  1280x720    15    │   98.46MiB  191k https │ vp09.00.31.08  191k video only          720p, webm_dash

holta commented 8 months ago

@deldesir mentions this might be relevant?

yt-dlp/yt-dlp#6078

holta commented 8 months ago

Just FYI testing for yt-dlp's exit code (error code) is straightforward — failures such as the inability to download these 2 videos are at least correctly reported (i.e. exit code 1) — as confirmed by:

root@box:~# yt-dlp --format best https://youtu.be/ZfbiLfSczgA
...
root@box:~# echo $?
1

And in situations when a video is successfully and completely downloaded, that too is correctly reported (i.e. exit code 0) — as confirmed by:

root@box:~# yt-dlp https://youtu.be/ZfbiLfSczgA
...
root@box:~# echo $?
0

root@box:~# yt-dlp --version
2023.12.30

chapmanjacobd commented 8 months ago

interesting... both --format best and --format-sort 'tbr~1000' for that URL are using format 22 which is apparently broken: https://github.com/yt-dlp/yt-dlp/issues/9071

iiab / calibre-web

2 videos "failed to download: 'NoneType' object is not subscriptable" — yt-dlp `--format best` fails to download complete video (same 1503232-byte .mp4.part fragment every time!) #135

114