search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.75k stars 10.07k forks source link

Slow extraction speed for Youtube video urls #32202

Open dhwz opened 1 year ago

dhwz commented 1 year ago

Question

I'm using youtube-dl only to extract video urls from youtube, it's extremely slow (takes up to 30secs to extract the urls) for a while now. I've also checked the open issues, one was talking about that but the final conclusion was it won't be fixed and yt-dlp could be used. I agree yt-dlp is much faster (takes maybe 3sec to get a URL), any chance we can get such modifications used on yt-dlp also on youtube-dl? yt-dlp is no alternative for me as I'm fixed to a system which still uses Python 2.7

dirkf commented 1 year ago

Feel free to submit a PR that makes the extraction process faster.

What takes the time now is that YT makes the extractor execute some junk JavaScript to obtain media links that are not throttled, though I took ~6% off this in a recent mod.

The extractor could provide you with undecoded media links but then the download speed will be throttled instead. The current extraction process aims to get the widest combination of unthrottled formats. If a format that is seen in the browser isn't being offered by yt-dl, it seems to provoke complaints.

yt-dlp uses other tactics that are either impractical or beyond the interest/capability of any contributor:

HPP8140 commented 1 year ago

any option to prefer throttled links for extraction speed?

HPP8140 commented 6 months ago

any option to prefer throttled links for extraction speed?

dirkf commented 6 months ago

Not without modifying the code. Also, one part of the JS execution is required when YT sends an encoded media link (signature) to decode the signature, without which you have no media URL, throttled or otherwise.

As always, a full refund of 0 0-based monetary units is offered if you find the program unsatisfactory.

HPP8140 commented 6 months ago

Not without modifying the code. Also, one part of the JS execution is required when YT sends an encoded media link (signature) to decode the signature, without which you have no media URL, throttled or otherwise.

As always, a full refund of 0 0-based monetary units is offered if you find the program unsatisfactory.

Okay, appreciate the response, thanks for closing the loop on this.

I will keep using the old verson of code, until someting major breaks that I am unable to understand how to manually patch because that part of the code changed so drastically among multiple files LOL. Then I will have to find time to update my SoC to be able to run modern versions of the code on Python 3 etc.

dirkf commented 6 months ago

Recent yt-dl versions are running fine, though slowly, with Python 2.7.1 on a Broadcom 7405 SoC with 2xMIP4380 CPUs (oh, and invoking wget for each _request_webpage()). You just have to adjust expectations.

HPP8140 commented 6 months ago

Recent yt-dl versions are running fine, though slowly, with Python 2.7.1 on a Broadcom 7405 SoC with 2xMIP4380 CPUs (oh, and invoking wget for each _request_webpage()). You just have to adjust expectations.

Currently using a legacy PogoPlug E02 SoC running Debian jessie :) Need to upgrade to some current Odroid or Pi

So instead of <11 seconds for youtube_dl -g, it can take up to 2 minutes per link...so quite a big difference. After grabbing all links & metadata from my yt watch playlist, I forward links to pyLoad.

HPP8140 commented 5 months ago

Recent yt-dl versions are running fine, though slowly, with Python 2.7.1 on a Broadcom 7405 SoC with 2xMIP4380 CPUs (oh, and invoking wget for each _request_webpage()). You just have to adjust expectations.

Well looks like the time has come....it was a good run. Old Version 2021.12.17 without nsig :) no longer downloading (403)

python -m youtube_dl -v https://www.youtube.com/watch?v=crnDkGUdoew

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=crnDkGUdoew']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.16 (CPython) - Linux-3.14.0-kirkwood-tld-1-armv5tel-with-debian-8.10
[debug] exe versions: ffmpeg 4.1.4-1, ffprobe 4.1.4-1
[debug] Proxy map: {}
[youtube] crnDkGUdoew: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://rr5---sn-8xgp1vo-p5qy.googlevideo.com/videoplayback?expire=1718503537&ei=EfRtZprqHZ2LkucPq4iggA8&ip=96.255.39.226&id=o-AM3HxxJOOm5Ve13igdogFDC7B84GzF1H3NiNc3ga2Yvn&itag=137&aitags=134%2C136%2C137%2C160&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=KB&mm=31%2C26&mn=sn-8xgp1vo-p5qy%2Csn-ab5l6nkd&ms=au%2Conr&mv=m&mvi=5&pl=18&initcwndbps=1111250&bui=AbKP-1P-XlxvEGAgw9c9nOLxvl0L1MW7_fKyxhf-bG6kdTsfgejTfqaDcx2RgNH9wcp2M5qM8R0m_RSG&spc=UWF9f2TkcOed3XvUROmKS8L34hfTstUwPV-IcAlM0I9URngKM3s8vcHmlgMi&vprv=1&svpuc=1&mime=video%2Fmp4&ns=3kshX-WwBaduvXJu2wKIW-4Q&rqh=1&gir=yes&clen=843238223&dur=1563.366&lmt=1718471588148623&mt=1718481655&fvip=5&keepalive=yes&c=WEB&sefc=1&txp=7209224&n=ubc0lAQ-fTeNojJB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRAIgAyYWfgDeNSuJk6zwKp2_I3fJWW9scii8oFV6aMp-D5MCICfSI0fCMCtIDzh59Mpl5r3et8LxH7qCvrRxik8cIilY&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHlkHjAwRQIgFDmUYPolqFMl-anaT9e67R1pdDyL3zZtDAwR_LLmMMoCIQC6XqWFIvp1yr_jenzMuHBw-0CMf4jaiIV0909FK6mGcg%3D%3D'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/lib/python2.7/youtube_dl/YoutubeDL.py", line 1970, in process_info
    partial_success = dl(fname, new_info)
  File "/usr/lib/python2.7/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/usr/lib/python2.7/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python2.7/youtube_dl/downloader/http.py", line 351, in real_download
    establish_connection()
  File "/usr/lib/python2.7/youtube_dl/downloader/http.py", line 116, in establish_connection
    raise err
HTTPError: HTTP Error 403: Forbidden
dirkf commented 5 months ago

Use the nightly build (#30839, Known Issues) and report if that fails. The old release has probably not worked in many cases for several years.

HPP8140 commented 5 months ago

Thanks yeah nightly build works, but then stopped working yesterday, but now looks like it's working again. Need to hurry up and set-up a faster SOC.

dirkf commented 5 months ago

This broke 2021.12.17. The nightly build was working until YT player 590f65a6 and then started working again when that player was replaced by 84314bef.