ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.23k stars 10.03k forks source link

Youtube live stream download ends after 6 hours #20516

Open 4ndriH opened 5 years ago

4ndriH commented 5 years ago

I have been downloading live streams on YouTube and noticed that it always ends after 6 hours (+/- a few seconds). At first I thought its something with youtube-dl but after recording it with VLC I got the same result. I looked through the help article but could not find anything related to this. Is there anything you can do about it? I am aware that this is not the most pressing issue there is because few people are downloading such long live streams (at least I think so) and it might prove difficult to find a solution for this because it seams to be something on YouTube's side.

Due to the immense code output I can not add much of value. I will try my best though.

C:\Users\XXX>youtube-dl -v https://www.youtube.com/watch?v=UVxU2HzPGug                                            
[debug] System config: []                                                                                               
[debug] User config: ['--no-mtime', '-o', 'C:/Users/XXX/Downloads/%(title)s.%(ext)s', '-f', 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4']                                                                                                  
[debug] Custom config: []                                                                                               
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=UVxU2HzPGug']                                        
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252                                                       
[debug] youtube-dl version 2019.03.18                                                                                   
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763                                                          
[debug] exe versions: ffmpeg N-92396-g55e021f39b, ffprobe N-92396-g55e021f39b                                           
[debug] Proxy map: {} 
[https @ 0000017b83195980] Opening 'https://r8---sn-nfpnnjvh-9ans.googlevideo.com/videoplayback/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/keepalive/yes/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8/sq/2269225/goap/clen%3D18585%3Blmt%3D1553951362306609/govp/clen%3D22709%3Blmt%3D1553951362306606/dur/2.000/file/seg.ts' for reading
[https @ 0000017b831eac00] Opening 'https://r8---sn-nfpnnjvh-9ans.googlevideo.com/videoplayback/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/keepalive/yes/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8/sq/2269226/goap/clen%3D19057%3Blmt%3D1553951362306617/govp/clen%3D20391%3Blmt%3D1553951362306613/dur/2.000/file/seg.ts' for reading
[https @ 0000017b83340e80] Opening 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/dover/11/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8' for reading
[https @ 0000017b831eac00] Opening 'https://r8---sn-nfpnnjvh-9ans.googlevideo.com/videoplayback/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/keepalive/yes/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8/sq/2269227/goap/clen%3D18861%3Blmt%3D1553951362306624/govp/clen%3D24287%3Blmt%3D1553951362306623/dur/2.000/file/seg.ts' for reading
[https @ 0000017b83340e80] Opening 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/dover/11/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8' for reading
[https @ 0000017b831eac00] Opening 'https://r8---sn-nfpnnjvh-9ans.googlevideo.com/videoplayback/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/keepalive/yes/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8/sq/2269228/goap/clen%3D19024%3Blmt%3D1553951362306632/govp/clen%3D23371%3Blmt%3D1553951362306633/dur/2.000/file/seg.ts' for reading
[https @ 0000017b83340e80] Opening 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/dover/11/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8' for reading
[https @ 0000017b829a69c0] HTTP error 403 Forbidden
[hls,applehttp @ 0000017b8222b240] keepalive request failed for 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/UVxU2HzPGug.1/itag/91/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D160/hls_chunk_host/r8---sn-nfpnnjvh-9ans.googlevideo.com/ei/mJSgXN7hDdOT1gKwiLewBQ/gcr/ch/playlist_type/DVR/initcwndbps/16760/mm/32/mn/sn-nfpnnjvh-9ans/ms/lv/mv/m/pcm2cms/yes/pl/45/dover/11/keepalive/yes/playlist_duration/30/manifest_duration/30/mt/1554027565/disable_polymer/true/ip/2a02:1205:503c:3f00:ec95:e57f:1a4b:f6eb/ipbits/0/expire/1554049272/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,ei,gcr,playlist_type,initcwndbps,mm,mn,ms,mv,pcm2cms,pl/signature/9C310477E509897A1866FF4CB32AF60A0689414C.01A6A51C9605FAD0CDFB9CFE4F854041E4729957/key/dg_yt0/playlist/index.m3u8', retrying with new connection: Server returned 403 Forbidden (access denied)
[https @ 0000017b832b69c0] HTTP error 403 Forbidden
[hls,applehttp @ 0000017b8222b240] Failed to reload playlist 0
[https @ 0000017b832b1880] HTTP error 403 Forbidden
[hls,applehttp @ 0000017b8222b240] Failed to reload playlist 0
frame=648060 fps= 30 q=-1.0 Lsize=  290429kB time=06:00:01.99 bitrate= 110.1kbits/s speed=   1x
video:151391kB audio:126574kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 4.483879%
[ffmpeg] Downloaded 297399166 bytes
[download] 100% of 283.62MiB in 05:59:59

That is the final output of the download. The part before "[https @ 0000017b829a69c0] HTTP error 403 Forbidden" fills the rest of my cmd window and is not of much use imo.

cheers

PS: In case the devs deem this unnecessary or unsolvable do not hesitate to close this Issue.

github-userx commented 5 years ago

My guess would be that youtube/google itself resets your token/connection after this amount of time.

4ndriH commented 5 years ago

My guess would be that youtube/google itself resets your token/connection after this amount of time.

Wouldn't this cause the stream you are watching on YT itself to stop as well? I had the stream open next to the CMD and it was still running there. Thinking about it, it might still be a token/connection thing with YT just refreshing it periodically. In that case it should be possible add a similar function to youtube-dl (maybe only when a stream is detected).

rautamiekka commented 5 years ago

My guess would be that youtube/google itself resets your token/connection after this amount of time.

Wouldn't this cause the stream you are watching on YT itself to stop as well? I had the stream open next to the CMD and it was still running there.

Nah. Like, even Google would think about that possibility, and stealthily reset.

FarmaanElahi commented 5 years ago

The token reset seems like a possibility here. YouTube might be refreshing the token after every 6 hour

DanGamingTV commented 4 years ago

I made a bash script to get a new m3u8 url when ffmpeg stops running. Feel free to use and adapt this to whatever you need.

#Explanation:
#
#function get() will take 3 paramaters: param 1 is the youtube uri, param 2 is the directory to download to, param 3 is the filename format (includes extension, such as .ts)
#get() then passes these args to function newurl(), along with an extra param which is a starting number for multiple files in the format 1test.ts, 2test.ts and so on. a new video file is created when ffmpeg stops (due to the m3u8 url expiring after 6 hrs)
#
#newurl() has 4 input params, 1 is the url, 2 is the directory, 3 is the file name, 4 is the number (from either get() or again())
#newurl() sets its own local vars from the params it gets, and adds one to the temp file number when it gets called again from again()
#newurl() then passes those params over to again()
#
#again() takes 5 params, 1 is the file number, 2 is the m3u8 stream url, 3 is the directory, 4 is the file name, 5 is the youtube video url (to be returned to newurl() when the stream url expires)
#
#Example usage: get "https://www.youtube.com/watch?v=21X5lGlDOfg" "C:\tmp" "NASA_TV.ts"

again() {
    tempnum=$1
    ffmpeg -i $2 -c copy $3$1$4 && newurl $5 $3 $4 $1 #ffmpeg downloads youtube stream to directory, then calls newurl() when ffmpeg stops
}
newurl() {
    local m3url=$(youtube-dl -f best -g $1) #store m3u8 stream url from youtube-dl as string
    local tempnum=$4 #file number, this is sent to again() so again() can send it back
    local tempdir=$2 #also sent to again so it can be sent back
    local tempfilename=$3 #also sent to again so it can be sent back
    let tempnum+=1 #add 1 to file number recieved from arguments
    again ${tempnum} ${m3url} ${tempdir} ${tempfilename} $1 #send required args to again(), and also send url so it can be sent back to newurl()
}
get() {
    newurl $1 $2 $3 0 #send args
}
#get "url" "directory" "filename"
4ndriH commented 4 years ago

As I have never used bash I do not understand how I get it to run. I do get "hello world" scripts to run but I am lost with the get command you have. Like do I just write: m3u8.sh get "URL" "dir" "file" or do I have to first enter the file and then the get command? Both do not seem to work for me. I am on a windows 10 machine if this has an impact.

DanGamingTV commented 4 years ago

Hi, you can run the script by installing git bash on windows. I haven't coded it to work with command line parameters, so you'll have to edit the bash file and change the command to what you need. When git bash is installed, .sh files will automatically be associated with it. Make sure you have ffmpeg installed and in your PATH.

4ndriH commented 4 years ago

Hi, Got it working. Thanks a lot for the script. It works like a charm!

Vozf commented 4 years ago

Same issue. I dont think that it's a url problem as i can continue downloading just by entering the same command after crash. This is a major issue and it is a shame that for more than a year nobody has fixed it.