search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.15k stars 10.02k forks source link

youtube-dl redirecting https to unsecure http using aria2c #20904

Closed fabricobjects2020 closed 5 years ago

fabricobjects2020 commented 5 years ago

Checklist

Verbose log

youtube-dl --verbose -f bestvideo+bestaudio --external-downloader aria2c --external-downloader-args "-s 8 -j 8 -x 8 -k 5M" https://www.youtube.com/watch?v=UY3mT-jKW6c

[debug] System config: [u'--prefer-free-formats']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-f', u'bestvideo+bestaudio', u'--external-downloader', u'aria2c', u'--external-downloader-args', u'-s 8 -j 8 -x 8 -k 5M', u'https://www.youtube.com/watch?v=UY3mT-jKW6c']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.24
[debug] Python version 2.7.5 (CPython) - Linux-5.0.10-1.el7.elrepo.x86_64-x86_64-with-redhat-7.6-Nitrogen
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6
[debug] Proxy map: {}
[youtube] UY3mT-jKW6c: Downloading webpage
[youtube] UY3mT-jKW6c: Downloading video info webpage
[youtube] {22} signature length 43.42, html5 player vflU6l_su
[youtube] {43} signature length 43.42, html5 player vflU6l_su
[youtube] {18} signature length 43.42, html5 player vflU6l_su
[youtube] {137} signature length 43.42, html5 player vflU6l_su
[youtube] {248} signature length 43.42, html5 player vflU6l_su
[youtube] {136} signature length 43.42, html5 player vflU6l_su
[youtube] {247} signature length 43.42, html5 player vflU6l_su
[youtube] {135} signature length 43.42, html5 player vflU6l_su
[youtube] {244} signature length 43.42, html5 player vflU6l_su
[youtube] {134} signature length 43.42, html5 player vflU6l_su
[youtube] {243} signature length 43.42, html5 player vflU6l_su
[youtube] {133} signature length 43.42, html5 player vflU6l_su
[youtube] {242} signature length 43.42, html5 player vflU6l_su
[youtube] {160} signature length 43.42, html5 player vflU6l_su
[youtube] {278} signature length 43.42, html5 player vflU6l_su
[youtube] {140} signature length 43.42, html5 player vflU6l_su
[youtube] {171} signature length 43.42, html5 player vflU6l_su
[youtube] {249} signature length 43.42, html5 player vflU6l_su
[youtube] {250} signature length 43.42, html5 player vflU6l_su
[youtube] {251} signature length 43.42, html5 player vflU6l_su
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on u'https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes'
[download] Destination: TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4
[debug] aria2c command line: aria2c -c -s 8 -j 8 -x 8 -k 5M --out 'TWICE TV '"'"'FANCY'"'"' EP.01-UY3mT-jKW6c.f137.mp4.part' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept-Language: en-us,en;q=0.5' --header 'Accept-Encoding: gzip, deflate' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0' '--check-certificate=true' -- 'https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes'

04/29 23:28:21 [NOTICE] Downloading 1 item(s)

04/29 23:28:21 [ERROR] CUID#7 - Download aborted. URI=https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes
Exception: [AbstractCommand.cc:351] errorCode=1 URI=https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AFYtGSyL7Xrcby1O3P38_GJF_YrHifNjw8MR83xpneJ5&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=08DHXIegN47BgwOO3pHABQ&initcwndbps=648750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594747&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616500&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=709028CB82F815F118BE69BC5563C59538A36C37.0CD9EADE5DEF270CB99A0C45E3516A7CCC9466D8&ratebypass=yes
  -> [SocketCore.cc:932] errorCode=1 SSL initialization failed: The request is invalid.

04/29 23:28:21 [NOTICE] Download GID#084da8bf92c54278 not complete: /home/devops23/Videos/TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
084da8|ERR |       0B/s|/home/devops23/Videos/TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4.part

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

ERROR: aria2c exited with code 1
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main
    _real_main(argv)
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2008, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 807, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 861, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1642, in process_video_result
    self.process_info(new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1909, in process_info
    partial_success = dl(fname, new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/external.py", line 64, in real_download
    self.get_basename(), retval))
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 624, in report_error
    self.trouble(error_message, tb)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 586, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

Description

Trying to download video from youtube using aria2c and axel. I found that both external downloaders are redirecting https to http - just started today. Aria2c did not tell me what was wrong (see above output) but here is the output from axel:

[devops23@devops-wks-01 Videos]$ youtube-dl -f bestvideo+bestaudio --external-downloader axel --external-downloader-args "-n 16" https://www.youtube.com/watch?v=UY3mT-jKW6c [youtube] UY3mT-jKW6c: Downloading webpage [youtube] UY3mT-jKW6c: Downloading video info webpage WARNING: Requested formats are incompatible for merge and will be merged into mkv. [download] Destination: TWICE TV 'FANCY' EP.01-UY3mT-jKW6c.f137.mp4 Initializing download: https://r5---sn-ab5szn7l.googlevideo.com/videoplayback?id=o-AL3RgL8yh5Oc0MXpMz6IppstrzS83yM175p-xesl_V2i&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&itag=137&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-ab5szn7l%2Csn-p5qlsnsd&ms=au%2Conr&mv=m&pl=21&ei=f8HHXPHxI4ne8wTz1qeQDw&initcwndbps=633750&mime=video%2Fmp4&gir=yes&clen=254011855&dur=749.048&lmt=1556541006991783&mt=1556594977&fvip=1&keepalive=yes&c=WEB&txp=4535432&ip=209.36.88.148&ipbits=0&expire=1556616671&sparams=ip%2Cipbits%2Cexpire%2Cid%2Caitags%2Csource%2Crequiressl%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cei%2Cinitcwndbps%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&key=yt8&signature=A416D5256C1E1B92F1B73F069ECBC955CE44FA86.2613DFA242A3B34096E35F8E2BA71CFF00EB054F&ratebypass=yes Unable to connect to server r5---sn-ab5szn7l.googlevideo.com:80

As you can see from the output google video is redirecting from https to unsecure http protocol which is not allowed outbound on my network.

dstftw commented 5 years ago

As clearly seen from the log youtube-dl passes https URL to external downloader. Everything that happens later is up to external downloader.

fabricobjects2020 commented 5 years ago

[RESOLVED]:

I uninstalled aria2c via yum then git cloned aria2 from github repository then recompiled with following options: $ cd /tmp $ git clone [paste aria2 git URL here] $ cd aria2 $ sudo autoreconf -iv $ sudo ./configure --without-gnutls --with-openssl $ sudo make -j 4 $ cd src $ cp ./aria2c /usr/bin

Apparently, there is a problem with aria2c and gnuTLS. You have to recompile aria2c to disable gnuTLS and enable OpenSSL.

[ NOTE: ] IF using Fedora, Centos, or RHEL you have to yum install the following dependencies:

$ sudo yum install gcc gcc-c++ kernel-ml-devel libgcrypt-devel libxml2-devel openssl-devel gettext-devel cppunit autoconf automake libtool nasm