Panopto huge fragment 1

jxu commented 2 years ago

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.03.08.1. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region

US

Description

This is a bug report on the abnormal behavior I observed in the comments of the original site request #1946 @coletdjnz

When downloading, fragment 1 is huge. For example, of a 1.9 GB video, I recorded fragment 1 as 633 MB.

The reason I suspect it is a bug is because if I interrupt downloading at say fragment 300 and then resume, yt-dlp will start writing to video.mp4.part-Frag1.part which again swells to a massive size instead of the leftover fragment videomp4.part-Frag300.part. This causes the final video to be much larger than needed.

Also attached is m3u8 files I saw used in firefox network dev tools, renamed for clarity.

m3u8s.zip

Verbose log

[debug] Command-line config: ['-vU', '--keep-fragments', '--cookies', 'cookies.txt', 'https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=88ac0972-bbce-4aaa-9e55-ae59011f5471']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.03.08.2 [e491d06d3]
[debug] Python version 3.8.10 (CPython 64bit) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: brotli, Cryptodome, secretstorage, mutagen, sqlite, websockets
[debug] Proxy map: {'no': 'localhost,127.0.0.0/8,::1'}
Latest version: 2022.03.08.1, Current version: 2022.03.08.2
yt-dlp is up to date (2022.03.08.2)
[debug] [Panopto] Extracting URL: https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=88ac0972-bbce-4aaa-9e55-ae59011f5471
[Panopto] 88ac0972-bbce-4aaa-9e55-ae59011f5471: Downloading JSON metadata
[Panopto] 88ac0972-bbce-4aaa-9e55-ae59011f5471: Downloading m3u8 information
[Panopto] 88ac0972-bbce-4aaa-9e55-ae59011f5471: Downloading m3u8 information
[Panopto] 88ac0972-bbce-4aaa-9e55-ae59011f5471: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 88ac0972-bbce-4aaa-9e55-ae59011f5471: Downloading 1 format(s): 1168
[debug] Invoking downloader on "https://d2y36twrtb17ty.cloudfront.net:443/sessions/36fb7cb1-6093-41e5-a4d3-ae59011f546a/88ac0972-bbce-4aaa-9e55-ae59011f5471-b41b189c-c211-49bd-9d8b-ae5901639190.hls/1164592/index.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 482
[download] Destination: Lecture 14 [88ac0972-bbce-4aaa-9e55-ae59011f5471].mp4
[download]   0.0% of ~284.13GiB at  2.64MiB/s ETA 30:34:07

coletdjnz commented 2 years ago

https://github.com/yt-dlp/yt-dlp/issues/2001 might be related

I can reproduce this too, even on git master. I'll see if I can find a public video

coletdjnz commented 2 years ago

format 644: https://sunwayedu.ap.panopto.com/Panopto/Pages/Viewer.aspx?id=f4ac9462-de97-4fca-a375-ae2a0023e5df format 457: https://gauchocast.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=0d29c48b-41d0-4a83-91a0-ae510152433a

Lesmiscore commented 2 years ago

That seems to be a bit different issue than #2001, as it happens on initialization fragment. (2001 is at media fragments without initialization ones)

https://github.com/yt-dlp/yt-dlp/blob/1fb707badb35d01d4ad7831a19f3469de2ef9141/yt_dlp/downloader/hls.py#L188-L219

Here's first 10 lines of segment playlist in question:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="fragmented.mp4",BYTERANGE="1300@0"
#EXTINF:9.000111,
#EXT-X-BYTERANGE:516680@1300
fragmented.mp4

jxu commented 2 years ago

I noticed also GNOME Files / Nautilus reports the wrong duration in their Audio/Video properties tab (not sure where they get this metadata), reporting the duration of one fragment of just 9 seconds, and ffprobe reports double the bitrate (maybe calculated from the large fragment 1). For the video that was interrupted, ffprobe reports

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x560d108ca9c0] Found duplicated MOOV Atom. Skipped it

Lesmiscore commented 2 years ago

e4fa34a13e9f94f27f0fccae6bcadc8dd1ea1415 ?

Lesmiscore commented 2 years ago

remainder: was it fixed?

coletdjnz commented 2 years ago

Did some testing, seems like it is fixed. Doesn't get stuck on fragment 1 anymore and seems to detect the correct size of it.

yt-dlp / yt-dlp