search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.47k stars 9.96k forks source link

When downloading YouTube live video, download from very beginning #21255

Open fireattack opened 5 years ago

fireattack commented 5 years ago

Checklist

Description

Currently, when downloading an ongoing live stream video on YouTube, youtube-dl will start from the current time.

I was wondering if we can add a way to download it from the very beginning instead. (I can confirm it is technical-wise possible at least, the video sources are available in m3u8 files etc.).

This way, the user can get the full video once the stream is finished. Otherwise, at least the first few seconds will be missing since you can't really start downloading before the live starts.

Of course, you can start the download when the live is done, which typically will solve this problem. However, some uploaders delete their live stream immediately after premiere, makes it somewhat difficult (also if you wait too long, you can't get HLS version any more, only re-compressed version.)

Thanks.

fireattack commented 5 years ago

I did a quick test myself again.

  1. I use webcam to start a quick live stream (by using https://www.youtube.com/livestreaming/) at https://www.youtube.com/watch?v=12REdrcaP1w.
  2. I waited for 1-2 min and then start to download it with youtube-dl.
  3. I waited a few minutes, and then terminated my stream. The youtube-dl download session ended later automatically.

Part of the log:

[hls,applehttp @ 00000000004c3500] Opening 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/293/goap/clen%3D22961%3Blmt%3D1559255013176245/govp/clen%3D64154%3Blmt%3D1559255013176251/dur/1.000/file/seg.ts' for reading
[https @ 0000000003c5be00] Opening 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8' for reading
[https @ 00000000039345c0] Cannot reuse HTTP connection for different host: r5---sn-q4fl6nlz.googlevideo.com:-1 != r1---sn-n0qqxoapo3-jaal.googlevideo.com:-1
[hls,applehttp @ 00000000004c3500] keepalive request failed for 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/294/goap/clen%3D22981%3Blmt%3D1559255013176256/govp/clen%3D64017%3Blmt%3D1559255013176258/dur/1.000/file/seg.ts', retrying with new connection: Invalid argument
[hls,applehttp @ 00000000004c3500] Opening 'https://r1---sn-n0qqxoapo3-jaal.googlevideo.com/videoplayback/id/12REdrcaP1w.0/itag/96/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D137/hls_chunk_host/r1---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/V1jwXJvgFM_5igSbkIaYAw/gcr/us/initcwndbps/12270/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/keepalive/yes/mt/1559255077/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559276727/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/4A019262A7752F272786E5CA8B0A65680D1B855C.3E1B47BD68DB818609D3E6C57B3DE4FD350640DF/key/dg_yt0/playlist/index.m3u8/sq/294/goap/clen%3D22981%3Blmt%3D1559255013176256/govp/clen%3D64017%3Blmt%3D1559255013176258/dur/1.000/file/seg.ts' for reading
frame= 5400 fps= 30 q=-1.0 Lsize=   13487kB time=00:02:59.97 bitrate= 613.9kbits/s speed=1.01x
video:10515kB audio:2870kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.756449%
[ffmpeg] Downloaded 13810230 bytes
[download] 100% of 13.17MiB in 02:59

The video file I downloaded is only 3min:

General
Complete name                            : D:\test-12REdrcaP1w.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 13.2 MiB
Duration                                 : 3 min 0 s
Overall bit rate                         : 614 kb/s
Writing application                      : Lavf58.20.100

The stream, if recorded from very beginning, should be around 4:55 as you can see in above link.

fireattack commented 5 years ago

Also by simply looking at the network traffic, you can see youtube-dl will only fetch the new video segments everytime one comes out (instead of trying to catch up all the available segments from the very beginning, which would have a much higher download speed).

fireattack commented 5 years ago

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard.

This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.

ealgase commented 5 years ago

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard.

This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.

Cancelling with CTRL+C works for me.

fireattack commented 5 years ago

Another thing I want to mention is that if you stop the download manually by ctrl-C it will generate a broken (merged) mp4 file. So you have to wait for the stream to end itself, which makes testing hard. This doesn't seem to happen if you download a normal video from YouTube: you can always interrupt the download and the part file (after rename) can be played just fine.

Cancelling with CTRL+C works for me.

Did you try on a ongoing live stream? It works on normal video but not live stream here.

I just tried again with this (currently ongoing): https://www.youtube.com/watch?v=QWTNIDZw608

The result is the same. I'm on Windows 7 64 bit with newest FFMPEG.

Logs

D:\>youtube-dl https://www.youtube.com/watch?v=QWTNIDZw608 -v
[debug] System config: []
[debug] User config: ['--write-description', '--write-info-json', '-i', '-4', '-o', '%(upload_date)s %(title)s-%(extractor)s-%(id)
s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=QWTNIDZw608', '-v']
[debug] Encodings: locale cp936, fs mbcs, out cp936, pref cp936
[debug] youtube-dl version 2019.05.20
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[youtube] QWTNIDZw608: Downloading webpage
[youtube] QWTNIDZw608: Downloading video info webpage
[youtube] QWTNIDZw608: Downloading m3u8 information
[youtube] QWTNIDZw608: Downloading MPD manifest
[debug] Default format spec: best/bestvideo+bestaudio
[info] Writing video description to: 20190603 Euronews English Live-youtube-QWTNIDZw608.description
[info] Writing video description metadata as JSON to: 20190603 Euronews English Live-youtube-QWTNIDZw608.info.json
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/QWTNIDZw608.0/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D136/hls_chunk_host/r4---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/rnf1XN7gL8PADvW3ulg/gcr/us/initcwndbps/14880/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559590708/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559612430/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/8B4CA4AFEE8D643727BF21EDE5097841DD9E61A7.87DE91740939E27D5DB59FF0F9C372C3ECBE9052/key/dg_yt0/playlist/index.m3u8'
[download] Destination: 20190603 Euronews English Live-youtube-QWTNIDZw608.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
" -i "https://manifest.googlevideo.com/api/manifest/hls_playlist/id/QWTNIDZw608.0/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/goi/160/sgoap/gir%3Dyes%3Bitag%3D140/sgovp/gir%3Dyes%3Bitag%3D136/hls_chunk_host/r4---sn-n0qqxoapo3-jaal.googlevideo.com/playlist_type/DVR/ei/rnf1XN7gL8PADvW3ulg/gcr/us/initcwndbps/14880/mm/44/mn/sn-n0qqxoapo3-jaal/ms/lva/mv/m/pl/22/dover/11/keepalive/yes/mt/1559590708/disable_polymer/true/ip/75.111.128.184/ipbits/0/expire/1559612430/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,goi,sgoap,sgovp,hls_chunk_host,playlist_type,ei,gcr,initcwndbps,mm,mn,ms,mv,pl/signature/8B4CA4AFEE8D643727BF21EDE5097841DD9E61A7.87DE91740939E27D5DB59FF0F9C372C3ECBE9052/key/dg_yt0/playlist/index.m3u8" -c copy -f mp4 "file:20190603 Euronews English Live-youtube-QWTNIDZw608.mp4.part"
[ffmpeg] Interrupted by user

[ffmpeg] Downloaded 4194352 bytes

[download] 100% of 4.00MiB in 00:13

Resulted video: 20190603 Euronews English Live-youtube-QWTNIDZw608.zip

(Unplayable on any player I have, including MPV and MPC-HC).

ealgase commented 5 years ago

Try it on Linux?

fireattack commented 5 years ago

I don't use Linux, but I may give it a try later in VM.

fireattack commented 5 years ago

Try it on Linux?

Hi, just tested in a Ubuntu 19 VM. Ctrl+C indeed doesn't generate broken file like on Windows (tested on Win 7 and Win 10).

But the main issue remains: it doesn't download from the very beginning on Linux either.

ealgase commented 5 years ago

Yeah, it doesn't download from the beginning, I was just troubleshooting the other issue.

Bangaio65 commented 5 years ago

I don't think you can do it with just youtube-dl, but you can using ffmpeg.

Caveats:

Instructions:

  1. Grab the HLS manifest url using youtube-dl (add -f best if you want the best quality) youtube-dl --no-warnings -f "best" -g -- <youtube-url>

  2. Figure out how many seconds is each segment (it's usually 2 or 5) and convert seconds to segments. If you're familiar with video editing, this is the time between keyframes. Say you want to go back 1 hour and the segment duration is 2 seconds. Then you do (1 60 60) / 2 = 1800. You need to "go back" 1800 segments on the stream. If you just need to go back to the start of the stream, just input a large number on the next step.

  3. Download using ffmpeg

    • add -t <seconds or HH:MM:SS> to specify a duration
    • it's recommended to save as .ts just in case the connection dies in the middle and you're left with an unusable .mp4 ffmpeg -live_start_index -1800 -i <m3u8_url> -t 600 -c copy livestream01.ts or ffmpeg -live_start_index -1800 -i <m3u8_url> -t 600 -bsf:a aac_adtstoasc -c copy livestream01.mp4

Hope this helps.

nixklai commented 3 years ago

I believe it is a ffmpeg-related issue, as I have marked the -live_start_index with an impossible number (-180000) and the output file still starts at the time of command execution.

For those whose debugging this issue: --hls-use-mpegts --ouput "%(title)s-%(id)s.ts" option helps turn the output into a .ts file for testing.

EDIT: Included output log EDIT 2: It seems like an old problem that ffmpeg team refuses to fix

nixklai commented 3 years ago

The issue seems to be caused by the initial responses from YouTube are truncated to only 5 segments, like in this test.m3u8 file I download via cURL.

.

johnhawkinson commented 3 years ago

I had meant to comment last month and was distracted:

EDIT 2: It seems like an old problem that ffmpeg team refuses to fix

I don't think this is quite accurate, or at least, it is more nuanced than described. The bug report you cite, https://trac.ffmpeg.org/ticket/5811, claims:

ffmpeg has an option to set the segment number where it must start reading. ... But this option is totally broken, and ffmpeg always starts at the third last segment.

I concur with the analysis of

  • resolution: => worksforme

That is, I use -live_start_index N with ffmpeg on a somewhat regular basis and it wors fine for me. I'm sure there's a reason @smalukav was having a problem, but it's not just broken all the time. I somehow doubt that's the failure with Youtube, but that's rank speculation on my part.

In any case, it doesn't seem like the ffmpeg developers are "refusing" — they report they cannot reproduce the problem and need a better reproduction case. That's a very different hing.

luke-jr commented 2 years ago

Can confirm that @Bangaio65 's instructions do not work