search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
129.83k stars 9.79k forks source link

[downloader/hlsnative] Resuming encrypted HLS streams leads to broken files #11015

Open shirishag75 opened 7 years ago

shirishag75 commented 7 years ago 
[$] youtube-dl --hls-prefer-native http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444                 
[HotStar] 1000153444: Downloading JSON metadata
[HotStar] 1000153444: Downloading TABLET JSON metadata
[HotStar] 1000153444: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 416
[download] Destination: The Promiscuous Wives-1000153444.mp4

Now twice the video has been downloaded but each time I have not been able to view the video either under mpv or vlc but when it is in .part I can play the video. It is only when it is complete, it doesn't play. I suspect some sort of corruption happening.

This is the ouput I get while viewing the .part file via ffprobe -

[$] ffprobe The\ Promiscuous\ Wives-1000153444.mp4.part                                                                    
ffprobe version 3.1.4-1 Copyright (c) 2007-2016 the FFmpeg developers
  built with gcc 6.2.0 (Debian 6.2.0-6) 20161010
  configuration: --prefix=/usr --extra-version=1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --enable-shared --disable-libtesseract --disable-stripping --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librubberband --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-frei0r --enable-chromaprint --enable-libopencv --enable-libx264
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[h264 @ 0x1693e00] non-existing SPS 0 referenced in buffering period
[h264 @ 0x1693e00] SPS unavailable in decode_picture_timing
[h264 @ 0x1693e00] non-existing SPS 0 referenced in buffering period
[h264 @ 0x1693e00] SPS unavailable in decode_picture_timing
[mpegts @ 0x168f3e0] PES packet size mismatch
Input #0, mpegts, from 'The Promiscuous Wives-1000153444.mp4.part':
  Duration: 00:11:12.00, start: 0.100667, bitrate: 2144 kb/s
  Program 1 
    Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 60 kb/s

Look forward to know more.

dstftw commented 7 years ago

Both work fine for me.

shirishag75 commented 7 years ago

it doesn't work at my end. Especially if for some reasons it stops, I have to start at beginning otherwise I just get - 

[$] mediainfo The\ Promiscuous\ Wives-1000153444.mp4.part                                                                       
General
Complete name                            : The Promiscuous Wives-1000153444.mp4.part
File size                                : 558 MiB

What it should give -

[$] mediainfo The\ Promiscuous\ Wives-1000153444.mp4.part                                                                         
General
ID                                       : 1 (0x1)
Complete name                            : The Promiscuous Wives-1000153444.mp4.part
Format                                   : MPEG-TS
File size                                : 543 MiB
Duration                                 : 35 min 23 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 2 143 kb/s

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L3.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 1 frame
Format settings, GOP                     : M=1, N=25
Codec ID                                 : 27
Duration                                 : 35 min 24 s
Bit rate mode                            : Constant
Nominal bit rate                         : 2 000 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.087

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format version                           : Version 4
Format profile                           : LC
Muxing mode                              : ADTS
Codec ID                                 : 15
Duration                                 : 35 min 23 s
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 spf)
Compression mode                         : Lossy

This was there sometime back, you can see the difference yourself.

bobbybluff commented 7 years ago

I am also experiencing this issue. Downloading a youtube video and attempting to play this on VLC Ubuntu will only have a black screen and no audio. However, when using the -k option (keep), both the audio (.mp4) and video (.webm) files play. I assume the error lays somewhere during combining of the files.

yan12125 commented 7 years ago

@bobbybluff I guess that's because you're using avconv. Try to install ffmpeg and run youtube-dl with --prefer-ffmpeg. If things are still broken, paste the output of youtube-dl -v --prefer-ffmpeg "your URL"

@shirishag75 Which player are you using? It's most likely player's problem on unable to handle mpeg-ts segments. If you have no choice on which player to use, install ffmpeg and replace --hls-prefer-native with --hls-prefer-ffmpeg from your commands.

bobbybluff commented 7 years ago

@yan12125 Thank you for your reply. I attempted using ffmpeg but still encounter same issue. Here's the output:

youtube-dl -v --prefer-ffmpeg https://www.youtube.com/watch?v=QAa2O_8wBUQ [debug] System config: [] [debug] User config: [] [debug] Command-line args: [u'-v', u'--prefer-ffmpeg', u'https://www.youtube.com/watch?v=QAa2O_8wBUQ'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2016.11.08.1 [debug] Python version 2.7.6 - Linux-3.13.0-53-generic-x86_64-with-Ubuntu-14.04-trusty [debug] exe versions: ffmpeg 3.2, ffprobe 3.2, rtmpdump 2.4 [debug] Proxy map: {} [youtube] QAa2O_8wBUQ: Downloading webpage [youtube] QAa2O_8wBUQ: Downloading video info webpage [youtube] QAa2O_8wBUQ: Extracting video information [youtube] QAa2O_8wBUQ: Downloading MPD manifest WARNING: Requested formats are incompatible for merge and will be merged into mkv. [debug] Invoking downloader on u'https://r6---sn-n4v7kn7s.googlevideo.com/videoplayback?id=4006b63bff300544&itag=137&source=youtube&requiressl=yes&usequic=no&mv=m&ms=au&nh=IgpwcjAxLnNqYzA3Kgw3Mi4xNC4yMjAuMTA&mn=sn-n4v7kn7s&mm=31&pl=19&initcwndbps=1046250&ratebypass=yes&mime=video/mp4&gir=yes&clen=70470272&lmt=1439629639453652&dur=380.040&signature=35B7375B8BB7EC5E6E518B882938405D13A653DD.5F67DF5198C0A69798E802BA4372E725C3EC597B&upn=nNajwXz_G3w&mt=1479012812&key=dg_yt0&ip=71.94.80.171&ipbits=0&expire=1479034487&sparams=ip,ipbits,expire,id,itag,source,requiressl,usequic,mv,ms,nh,mn,mm,pl,initcwndbps,ratebypass,mime,gir,clen,lmt,dur' [download] Destination: What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f137.mp4 [download] 100% of 67.21MiB in 00:29 [debug] Invoking downloader on u'https://r6---sn-n4v7kn7s.googlevideo.com/videoplayback?id=4006b63bff300544&itag=251&source=youtube&requiressl=yes&usequic=no&mv=m&ms=au&nh=IgpwcjAxLnNqYzA3Kgw3Mi4xNC4yMjAuMTA&mn=sn-n4v7kn7s&mm=31&pl=19&initcwndbps=1046250&ratebypass=yes&mime=audio/webm&gir=yes&clen=7125999&lmt=1449568716498724&dur=380.041&signature=857A8E46293471C759F401919F732A6BACB12887.7016012B0FE440A9995C695C4F2BD3A00BAD94AB&upn=nNajwXz_G3w&mt=1479012812&key=dg_yt0&ip=71.94.80.171&ipbits=0&expire=1479034487&sparams=ip,ipbits,expire,id,itag,source,requiressl,usequic,mv,ms,nh,mn,mm,pl,initcwndbps,ratebypass,mime,gir,clen,lmt,dur' [download] Destination: What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f251.webm [download] 100% of 6.80MiB in 00:02 [ffmpeg] Merging formats into "What is Dark Matter and Dark Energy-QAa2O_8wBUQ.mkv" [debug] ffmpeg command line: ffmpeg -y -i 'file:What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f137.mp4' -i 'file:What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f251.webm' -c copy -map 0:v:0 -map 1:a:0 'file:What is Dark Matter and Dark Energy-QAa2O_8wBUQ.temp.mkv' Deleting original file What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f137.mp4 (pass -k to keep) Deleting original file What is Dark Matter and Dark Energy-QAa2O_8wBUQ.f251.webm (pass -k to keep)

yan12125 commented 7 years ago

Just tried it and VLC 2.2.4 plays fine for me.

bobbybluff commented 7 years ago

Just tried it with SMPlayer and it works. Issue lies with VLC 2.2.1.

yan12125 commented 7 years ago

OK let's wait for @shirishag75 and see what's his problem

shirishag75 commented 7 years ago

hi all,

I was able to download it but after many tries. I am on Debian testing/strech and avconv is no longer an option. It is all ffmpeg now -

[$] apt-cache policy ffmpeg                                                                                                       
ffmpeg:
  Installed: 7:3.2-2
  Candidate: 7:3.2-2
  Version table:
 *** 7:3.2-2 600
        600 http://httpredir.debian.org/debian testing/main amd64 Packages
          1 http://httpredir.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status

The core issue is that re-downloading the video from where it stopped results in no play of the video. I use mpv as my preferred movie player as it doesn't take much memory, is good with kbd shortcuts and is out of the way -


─[$] apt-cache policy mpv                                                                                                           
mpv:
  Installed: 0.21.0-2
  Candidate: 0.21.0-2
  Version table:
 *** 0.21.0-2 600
        600 http://httpredir.debian.org/debian testing/main amd64 Packages
          1 http://httpredir.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status

Hope this info. clears the issues a little bit.

yan12125 commented 7 years ago

@shirishag75 First please paste the output of this command:

youtube-dl -v --hls-prefer-native http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444

And can you describe more on "can't play"? I tried a 3.1MB partial video with mpv 0.21.0 and it plays fine. By the way, mediainfo (version 0.7.90) shows correct info even for partial files:

$ mediainfo The\ Promiscuous\ Wives-1000153444.mp4.part
General
ID                                       : 1 (0x1)
Complete name                            : The Promiscuous Wives-1000153444.mp4.part
Format                                   : MPEG-TS
File size                                : 3.10 MiB
Duration                                 : 11 s 960 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 2 168 kb/s

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L3.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 1 frame
Format settings, GOP                     : M=1, N=25
Codec ID                                 : 27
Duration                                 : 12 s 0 ms
Bit rate mode                            : Constant
Nominal bit rate                         : 2 000 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.087

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format version                           : Version 4
Format profile                           : LC
Muxing mode                              : ADTS
Codec ID                                 : 15
Duration                                 : 11 s 925 ms
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 spf)
Compression mode                         : Lossy

I guess there's something wrong in downloading.

By the way, what's your version of python-crypto? This package is required to download encrypted streams provided by hotstar with youtube-dl's builtin downloader.

shirishag75 commented 7 years ago

First things first, my version of python-crypto is -

[$] apt-cache policy python-crypto

python-crypto:
  Installed: 2.6.1-6+b1
  Candidate: 2.6.1-6+b1
  Version table:
 *** 2.6.1-6+b1 600
        600 http://httpredir.debian.org/debian testing/main amd64 Packages
          1 http://httpredir.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status

Now output of the command as you shared is -

[$] youtube-dl -v --hls-prefer-native http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444                
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'--hls-prefer-native', u'http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.11.08.1
[debug] Python version 2.7.12+ - Linux-4.7.0-1-amd64-x86_64-with-debian-stretch-sid
[debug] exe versions: ffmpeg 3.2-2, ffprobe 3.2-2, rtmpdump 2.4
[debug] Proxy map: {}
[HotStar] 1000153444: Downloading JSON metadata
[HotStar] 1000153444: Downloading TABLET JSON metadata
[HotStar] 1000153444: Downloading m3u8 information
[debug] Invoking downloader on u'https://staragvod3-vh.akamaihd.net/i/videos/lifeok/sain/1848/1000153444_,16,180,400,800,1300,2000,_STAR.mp4.csmil/index_5_av.m3u8?null=0&id=AgA7XwESaF++HJKOKFgtHeUu2o4ruJPu+o+S3G6amr73YpAFtCy0BsObpjDT8haNTbxoERwMuA6abA%3d%3d'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 416
[download] Destination: The Promiscuous Wives-1000153444.mp4
[download]   2.4% of ~635.50MiB at 123.80KiB/s ETA 01:30:00^C
ERROR: Interrupted by user

While I'm able to view the partial file. What happens is at the very end. Let me explain the way I see it -

when the --hls-prefer-native switch is used, it makes fragments/segments of the file. If I interrupt it, some fragments will remain on disk while others need to be downloaded. As shared above, it says 416 fragments are needed to assemble the whole file.

Now if I interrupt the downloading, say lights go out or anything, in perfect scenario, using --hls-prefer-native it should look at both the .part file as well as the fragments and if something is wrong/incorrect it should delete that and start downloading after checking all the other pieces are good or not. I do not know whether this checking happens or not. I have seen though that let's say if I delete or .part file gets deleted it is possible to build that through the fragments file. But the resultant media file when everything is downloaded and done is not playable/viewable.

Does this make some more sense ?

siddht4 commented 7 years ago

well for these are there are 416 pieces of the file (fragments) , the downloader needs each file in sequence when all the 416 pieces are completed so that it can stitch/append all the files. the issue you are facing is due to network instability (major issue in your and my country) so a piece of file gets partially downloaded so it fails to stitch properly, even though all fragments are there. So i would recommend is to download via an external downloader or first play it externally through vlc

shirishag75 commented 7 years ago

I understand what you are saying. I do follow that the 'stitching' process may have some issue/bug. I don't know which is the element where that part needs to be followed up . Any pointers will be good .

I don't understand partially though your last statement, the part about

or first play it externally through vlc

let's say there is a stiching issue. If it is, is there a way that vlc can fix it rather than using mpv ?

siddht4 commented 7 years ago

use the command

wget -O output-filename.mp4 $( youtube-dl -g http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444 )

yan12125 commented 7 years ago

@siddht1 encrypted hls streams are not that easy

shirishag75 commented 7 years ago

that is the point that I was trying to tell/share. Btw this again failed at my end -

┌─[shirish@debian] - [~/Videos] - [4513]
└─[$] youtube-dl -v --hls-prefer-native http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444             
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'--hls-prefer-native', u'http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.11.08.1
[debug] Python version 2.7.12+ - Linux-4.7.0-1-amd64-x86_64-with-debian-stretch-sid
[debug] exe versions: ffmpeg 3.2-2, ffprobe 3.2-2, rtmpdump 2.4
[debug] Proxy map: {}
[HotStar] 1000153444: Downloading JSON metadata
[HotStar] 1000153444: Downloading TABLET JSON metadata
[HotStar] 1000153444: Downloading m3u8 information
[debug] Invoking downloader on u'https://staragvod3-vh.akamaihd.net/i/videos/lifeok/sain/1848/1000153444_,16,180,400,800,1300,2000,_STAR.mp4.csmil/index_5_av.m3u8?null=0&id=AgA7XwESaF++HKmdKFiuCX7ukDaVHaqLmr+POapKdfur9OZCV+kxbTC1wMlQ6iWxjMby6eftCNw+Mg%3d%3d'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 416
[download] Destination: The Promiscuous Wives-1000153444.mp4
[download]  26.7% of ~638.23MiB at 114.31KiB/s ETA 45:56^C
ERROR: Interrupted by user

In vlc it just goes to a loop and gives [mp3 @ 0x7f7c4cc51300] Header missing

In mediainfo and mpv you get to know what the issue is -

┌─[shirish@debian] - [~/Videos] - [4525]
└─[$] mpv --mute The\ Promiscuous\ Wives-1000153444.mp4.part                                                                    
(null) requires an argument
Playing: The Promiscuous Wives-1000153444.mp4.part
Failed to recognize file format.

Exiting... (Errors when loading file)

and


└─[$] mediainfo The\ Promiscuous\ Wives-1000153444.mp4.part                                                                    
General
Complete name                            : The Promiscuous Wives-1000153444.mp4.part
File size                                : 170 MiB

As can be seen it is the 'stiching' part which is at fault somewhere.

yan12125 commented 7 years ago

@shirishag75 Confirmed the bug. Thanks for the detailed steps.

siddht4 commented 7 years ago

@yan12125 yes i saw that too
wget -O output-filename.mp4 $( youtube-dl -g http://www.hotstar.com/tv/savdhaan-india/363/the-promiscuous-wives/1000153444 ) --2016-11-13 22:56:43-- https://staragvod3-vh.akamaihd.net/i/videos/lifeok/sain/1848/1000153444_,16,180,400,800,1300,2000,_STAR.mp4.csmil/index_5_av.m3u8?null=0&id=AgAvHRkbMSwyH1OiKFhFGhPWwIog6Vxgvgu1KjBYpMA5RDDVcuCn%2fmozQdSBM9e1AQVDZ7ZbYHaOig%3d%3d Resolving staragvod3-vh.akamaihd.net (staragvod3-vh.akamaihd.net)... 49.44.50.56, 49.44.50.43, 49.44.50.49, ... Connecting to staragvod3-vh.akamaihd.net (staragvod3-vh.akamaihd.net)|49.44.50.56|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 101785 (99K) [application/vnd.apple.mpegurl] Saving to: ‘output-filename.mp4’

output-filename.mp4 100%[===================>] 99.40K 482KB/s in 0.2s

2016-11-13 22:56:45 (482 KB/s) - ‘output-filename.mp4’ saved [101785/101785]

so surely the problem is when retriving files , got the problem change the user agent to an android mobile phone , hotstar is blocking the stream

siddht4 commented 7 years ago

@shirishag75 even though its a bug. let me help you. eg : a file contains abcdefghjklmnopqrstuvwxyz so its 26 characters (refrence for pieces) i downloaded it as abcdefgghijklmnopqrstuvwxz

so the downloader sees that it has 26 characters and a,z are in front and back , mid is n is 13 so it stiches it as it thinks its the right pieces even though there is an extra g and there`s no y, but it cant confirm as there is not error detection . this is whats actually happening/occuring

siddht4 commented 7 years ago

@yan12125 is there anyway in youtube-dl where a checksum/crc can be done directly

shirishag75 commented 7 years ago

@siddht1 I got this, I am asking as for most people they would just give up. If there is no error-detection, then it is upto the developers to fix it and they can only do it if we highlight our issues and this is reproducible by them.

siddht4 commented 7 years ago

@shirishag75 i get you , and yes the developers can fix it , especially who are dealing for youtube-dl from several years. But i am to a developer (not for youtube-dl ), but for java calling of this program https://github.com/rg3/youtube-dl/issues/10975 , have faced issue as porting also needs work. this check can be done , so i can surely help if nedded. just checking crc/checksum can be done for an on line file

shirishag75 commented 7 years ago

Anybody working on this .... ping ?