search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.76k stars 10.08k forks source link

Support format fallback (was: ERROR: giving up after 10 retries) #3721

Open Elite opened 10 years ago

Elite commented 10 years ago 
youtube-dl -f 137+140 -v https://www.youtube.com/watch?v=USzA8aBaRNc
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', '137+140', '-v', 'https://www.youtube.com/watch?v=USzA8aBaRNc']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.10.1
[debug] Python version 2.7.3 - Linux-3.2.0-4-amd64-x86_64-with-debian-7.5
[debug] Proxy map: {}
[youtube] Setting language
[youtube] Confirming age
[youtube] USzA8aBaRNc: Downloading webpage
[youtube] USzA8aBaRNc: Downloading video info webpage
[youtube] USzA8aBaRNc: Extracting video information
[debug] Invoking downloader on u'https://r1---sn-aiglln7d.googlevideo.com/videoplayback?sver=3&nh=IgpwcjAxLmxocjA3KgkxMjcuMC4wLjE&gir=yes&mm=31&signature=BE80A45DD6089031F21DD3F38F3467391E96B3E0.58497AD778D0AE7F8DD467E4D8BDD09A6068E990&key=yt5&id=o-AD9cD6UbVG6OKDOp9GVs-sZZVw-paoVpe1UUFYGwjEhd&dur=273.273&ipbits=0&initcwndbps=11841000&mv=m&ms=au&ip=88.150.175.147&clen=130976906&sparams=clen%2Cdur%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmm%2Cms%2Cmv%2Cnh%2Crequiressl%2Csource%2Cupn%2Cexpire&mt=1410428969&source=youtube&lmt=1410169303233178&expire=1410450636&fexp=900223%2C901082%2C904843%2C905024%2C927622%2C931983%2C932404%2C932623%2C934024%2C934030%2C946023%2C947209%2C951900%2C953801&itag=137&requiressl=yes&upn=-wkNkGrw3fw&ratebypass=yes'
[download] Got server HTTP error. Retrying (attempt 1 of 10)...
[download] Got server HTTP error. Retrying (attempt 2 of 10)...
[download] Got server HTTP error. Retrying (attempt 3 of 10)...
[download] Got server HTTP error. Retrying (attempt 4 of 10)...
[download] Got server HTTP error. Retrying (attempt 5 of 10)...
[download] Got server HTTP error. Retrying (attempt 6 of 10)...
[download] Got server HTTP error. Retrying (attempt 7 of 10)...
[download] Got server HTTP error. Retrying (attempt 8 of 10)...
[download] Got server HTTP error. Retrying (attempt 9 of 10)...
[download] Got server HTTP error. Retrying (attempt 10 of 10)...
ERROR: giving up after 10 retries
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/downloader/http.py", line 66, in real_download
    data = self.ydl.urlopen(request)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1244, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 407, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 445, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
pricejn2 commented 10 years ago

The download url in your output appears to be invalid. However, it downloads fine for me from a different download url.

When I tried to add -g to your download command I get a Key Error, which may be related or a separate bug. 

youtube-dl -f 137+140 -v -g https://www.youtube.com/watch?v=USzA8aBaRNc
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', '137+140', '-v', '-g', 'https://www.youtube.com/watch?v=USzA8aBaRNc']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.16.1
[debug] Python version 2.7.3 - Linux-3.2.0-4-amd64-x86_64-with-debian-7.6
[debug] Proxy map: {}
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2014.09.16.1', 'console_scripts', 'youtube-dl')()
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/__init__.py", line 429, in main
    _real_main(argv)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/__init__.py", line 419, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1066, in download
    self.extract_info(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 535, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 577, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 844, in process_video_result
    self.process_info(new_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 885, in process_info
    self.to_stdout(info_dict['url'] + info_dict.get('play_path', ''))
KeyError: u'url'
Elite commented 10 years ago

The url https://www.youtube.com/watch?v=USzA8aBaRNc opens fine in web-browser.

juancri commented 10 years ago

This works perfectly fine for me:

$ youtube-dl -f 137+140 -v https://www.youtube.com/watch?v=USzA8aBaRNc [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['-f', '137+140', '-v', 'https://www.youtube.com/watch?v=USzA8aBaRNc'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2014.09.16 [debug] Python version 3.4.1 - Linux-3.16.2-1-ARCH-x86_64-with-arch [debug] Proxy map: {} [youtube] Setting language [youtube] Confirming age [youtube] USzA8aBaRNc: Downloading webpage [youtube] USzA8aBaRNc: Downloading video info webpage [youtube] USzA8aBaRNc: Extracting video information [debug] Invoking downloader on 'https://r5---sn-uxgg5-njak.googlevideo.com/videoplayback?mm=31&key=yt5&sver=3&upn=EjWHRyKTcHY&signature=D7B873C726CCE88E32ABC7561DBFB74BA320D4D4.3002D3013B0F09AD91766099506F42AA9AF63A79&clen=130976906&expire=1411039458&mt=1411017779&mv=m&initcwndbps=572500&ipbits=0&fexp=900064%2C927622%2C930666%2C931983%2C932404%2C934030%2C945242%2C946012%2C947209%2C952302%2C953801&ms=au&itag=137&lmt=1410169303233178&requiressl=yes&ip=190.22.227.96&source=youtube&dur=273.273&sparams=clen%2Cdur%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmm%2Cms%2Cmv%2Crequiressl%2Csource%2Cupn%2Cexpire&gir=yes&id=o-ADjvmZ9CxAIkRLw_I55Hu1_ikfVr3r56i5jik5ilzV_e&ratebypass=yes' [download] Destination: SHOW-YA - 「紅」～「HOWEVER」～「虹」-USzA8aBaRNc.f137.mp4 [download] 73.4% of 124.91MiB at 1.82MiB/s ETA 00:18^C

Can you try with the latest version?

yan12125 commented 8 years ago

The original example https://www.youtube.com/watch?v=USzA8aBaRNc now works fine. Here another example from #10310: https://www.youtube.com/watch?v=WPW4-JUIPys. As of writing, format 251 reports HTTP 500 error. In Firefox, the YouTube player first tries format 251, and then fallback to format 140 after the first failure. youtube-dl should support fallback, too.

gj12 commented 7 years ago

Same Problem here (with 2017.01.25):

youtube-dl -v  https://www.youtube.com/user/myst32YT
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/user/myst32YT']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.01.25
[debug] Python version 3.6.0 - Linux-4.7.7-x86_64-with-arch
[debug] exe versions: ffmpeg 3.2.2, ffprobe 3.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:user] myst32YT: Downloading channel page
[youtube:playlist] UUUeAg0dV9Xhauf0VF-CqbVg: Downloading webpage
[download] Downloading playlist: Uploads from myst32YT
[youtube:playlist] playlist Uploads from myst32YT: Downloading 57 videos
[download] Downloading video 1 of 57
[youtube] 9J4yF3OpYZQ: Downloading webpage
[youtube] 9J4yF3OpYZQ: Downloading video info webpage
[youtube] 9J4yF3OpYZQ: Extracting video information
[youtube] 9J4yF3OpYZQ: Downloading MPD manifest
[download] Use chemistry to remove the yellow from old plastic. [Not Retr0bright]-9J4yF3OpYZQ.mp4 has already been downloaded and merged
[download] Downloading video 2 of 57
[youtube] 4_W4FHM0SVU: Downloading webpage
[youtube] 4_W4FHM0SVU: Downloading video info webpage
[youtube] 4_W4FHM0SVU: Extracting video information
[youtube] 4_W4FHM0SVU: Downloading MPD manifest
[download] Lab Tips - A Simple Inexpensive Vacuum Chamber-4_W4FHM0SVU.mp4 has already been downloaded and merged
[download] Downloading video 3 of 57
[youtube] 3b8cuMpBcQ0: Downloading webpage
[youtube] 3b8cuMpBcQ0: Downloading video info webpage
[youtube] 3b8cuMpBcQ0: Extracting video information
[youtube] 3b8cuMpBcQ0: Downloading MPD manifest
[download] Lab Tips - Simple Water Condenser Setup-3b8cuMpBcQ0.mp4 has already been downloaded and merged
[download] Downloading video 4 of 57
[youtube] -Q9ow8pIa6g: Downloading webpage
[youtube] -Q9ow8pIa6g: Downloading video info webpage
[youtube] -Q9ow8pIa6g: Extracting video information
[youtube] -Q9ow8pIa6g: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Photo Chemistry - Silver Printing on Cotton Paper--Q9ow8pIa6g.mkv has already been downloaded and merged
[download] Downloading video 5 of 57
[youtube] ZqEWUw6sgpA: Downloading webpage
[youtube] ZqEWUw6sgpA: Downloading video info webpage
[youtube] ZqEWUw6sgpA: Extracting video information
[youtube] ZqEWUw6sgpA: Downloading MPD manifest
[download] Make Sodium peroxide-ZqEWUw6sgpA.mp4 has already been downloaded and merged
[download] Downloading video 6 of 57
[youtube] kqQSpRus-t0: Downloading webpage
[youtube] kqQSpRus-t0: Downloading video info webpage
[youtube] kqQSpRus-t0: Extracting video information
[youtube] kqQSpRus-t0: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Lab Tips - Back flow prevention when dissolving highly soluble gases-kqQSpRus-t0.mkv has already been downloaded and merged
[download] Downloading video 7 of 57
[youtube] 0hv4Qc6z8a0: Downloading webpage
[youtube] 0hv4Qc6z8a0: Downloading video info webpage
[youtube] 0hv4Qc6z8a0: Extracting video information
[youtube] 0hv4Qc6z8a0: Downloading MPD manifest
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r5---sn-4g5ednle.googlevideo.com/videoplayback?id=d21bf841ceb3f1ad&itag=134&source=youtube&requiressl=yes&mv=m&ms=au&mn=sn-4g5ednle&mm=31&initcwndbps=1950000&pl=42&nh=IgpwcjAxLmZyYTE2KgkxMjcuMC4wLjE&ratebypass=yes&mime=video/mp4&gir=yes&clen=3997196&lmt=1393619479050459&dur=108.776&mt=1485427220&key=dg_yt0&signature=3F44F48FCCEA7F412DA6EC24DFDD004D7526217F.18B2913E0662972D9148145D21031C0581AA5FA7&upn=HgOIJX7vHZE&ip=2a01:4f8:131:2365::2&ipbits=0&expire=1485448963&sparams=ip,ipbits,expire,id,itag,source,requiressl,mv,ms,mn,mm,initcwndbps,pl,nh,ratebypass,mime,gir,clen,lmt,dur'
[download] Buy OTC Nitrocellulose (Flash Paper)-0hv4Qc6z8a0.f134.mp4 has already been downloaded
[download] 100% of 3.81MiB
[debug] Invoking downloader on 'https://r5---sn-4g5ednle.googlevideo.com/videoplayback?id=d21bf841ceb3f1ad&itag=251&source=youtube&requiressl=yes&mv=m&ms=au&mn=sn-4g5ednle&mm=31&initcwndbps=1950000&pl=42&nh=IgpwcjAxLmZyYTE2KgkxMjcuMC4wLjE&ratebypass=yes&mime=audio/webm&gir=yes&clen=1879407&lmt=1411967165905285&dur=108.801&mt=1485427220&key=dg_yt0&signature=9644C0CD96360E6CA7038482E5EA10C41E5E3468.44B3C3DDC77F0802156D604E25DDD1D0859891DE&upn=HgOIJX7vHZE&ip=2a01:4f8:131:2365::2&ipbits=0&expire=1485448963&sparams=ip,ipbits,expire,id,itag,source,requiressl,mv,ms,mn,mm,initcwndbps,pl,nh,ratebypass,mime,gir,clen,lmt,dur'
[download] Got server HTTP error. Retrying (attempt 1 of 10)...
[download] Got server HTTP error. Retrying (attempt 2 of 10)...
[download] Got server HTTP error. Retrying (attempt 3 of 10)...
[download] Got server HTTP error. Retrying (attempt 4 of 10)...
[download] Got server HTTP error. Retrying (attempt 5 of 10)...
[download] Got server HTTP error. Retrying (attempt 6 of 10)...
[download] Got server HTTP error. Retrying (attempt 7 of 10)...
[download] Got server HTTP error. Retrying (attempt 8 of 10)...
[download] Got server HTTP error. Retrying (attempt 9 of 10)...
[download] Got server HTTP error. Retrying (attempt 10 of 10)...
ERROR: giving up after 10 retries
  File "/bin/youtube-dl", line 11, in <module>
    load_entry_point('youtube-dl==2017.1.25', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/__init__.py", line 444, in main
    _real_main(argv)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/__init__.py", line 434, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 1791, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 758, in process_ie_result
    extra_info=extra_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 866, in process_ie_result
    extra_info=extra)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 758, in process_ie_result
    extra_info=extra_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 1693, in process_info
    partial_success = dl(fname, new_info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/downloader/common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/downloader/http.py", line 131, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/downloader/common.py", line 162, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 568, in report_error
    self.trouble(error_message, tb)
  File "/usr/lib/python3.6/site-packages/youtube_dl-2017.1.25-py3.6.egg/youtube_dl/YoutubeDL.py", line 530, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())
xPMo commented 7 years ago

Video succeeds, audio fails (Internal Server Error 500):

youtube-dl 'https://www.youtube.com/watch/?v=-FGuaabGcYQ' --verbose 2>&1 | ix

output

arno01 commented 7 years ago

Same here with youtube-dl 2017.09.02 output

Upd today the download finished without an issue, might be a temporary issue from youtube side.

Oroneki commented 7 years ago

Here is my output. If the video was recently streamed, sometimes the "best" format isn't available for a while. When I choose another format the download starts and the error message 404 is gone. Browsers deal well with this issue choosing the next available format:

▶ youtube-dl KTUvvA9gH-s                                        
[youtube] KTUvvA9gH-s: Downloading webpage
[youtube] KTUvvA9gH-s: Downloading video info webpage
[youtube] KTUvvA9gH-s: Extracting video information
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[youtube] KTUvvA9gH-s: Downloading MPD manifest
WARNING: Your copy of avconv is outdated and unable to properly mux separate video and audio files, youtube-dl will download single file media. Update avconv to version 10-0 or newer to fix this.
ERROR: unable to download video data: HTTP Error 404: Not Found

~                                                                                                                                                                                                    ⍉
▶ youtube-dl KTUvvA9gH-s --list-formats    
[youtube] KTUvvA9gH-s: Downloading webpage
[youtube] KTUvvA9gH-s: Downloading video info webpage
[youtube] KTUvvA9gH-s: Extracting video information
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[info] Available formats for KTUvvA9gH-s:
format code  extension  resolution note
139          m4a        audio only DASH audio   52k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 27.27MiB
140          m4a        audio only DASH audio  132k , m4a_dash container, mp4a.40.2@128k (44100Hz), 72.82MiB
160          mp4        256x144    DASH video  116k , avc1.4d400c, 30fps, video only, 17.33MiB
133          mp4        426x240    DASH video  257k , avc1.4d4015, 30fps, video only, 32.58MiB
134          mp4        640x360    DASH video  659k , avc1.4d401e, 30fps, video only, 85.53MiB
135          mp4        854x480    DASH video 1212k , avc1.4d401f, 30fps, video only, 170.10MiB
136          mp4        1280x720   DASH video 2397k , avc1.4d401f, 30fps, video only, 320.84MiB
43           webm       640x360    medium , vp8.0, vorbis@128k
22           mp4        1280x720   hd720 , avc1.64001F, mp4a.40.2@192k (best)

~                                                                                                                                                                                                     
▶ youtube-dl -f 22 KTUvvA9gH-s         
[youtube] KTUvvA9gH-s: Downloading webpage
[youtube] KTUvvA9gH-s: Downloading video info webpage
[youtube] KTUvvA9gH-s: Extracting video information
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[youtube] KTUvvA9gH-s: Downloading MPD manifest
ERROR: unable to download video data: HTTP Error 404: Not Found

~                                                                                                                                                                                                    ⍉
▶ youtube-dl -f 140 KTUvvA9gH-s 
[youtube] KTUvvA9gH-s: Downloading webpage
[youtube] KTUvvA9gH-s: Downloading video info webpage
[youtube] KTUvvA9gH-s: Extracting video information
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[youtube] KTUvvA9gH-s: Downloading MPD manifest
[download] Destination: O É da Coisa, com Reinaldo Azevedo -01_12_2017-KTUvvA9gH-s.m4a
[download]  20.3% of 72.82MiB at  1.52MiB/s ETA 00:38^C
Fabian42 commented 5 years ago

What's the current state of this issue? Is this at least confirmed by a contributor?

Fabian42 commented 5 years ago

Also, how were I supposed to find this title, @dstftw?

Hrxn commented 5 years ago

Yes, has been confirmed by a contributor in the past, why?

proton1k commented 4 years ago

Why the format 22 fails with this error while for example format 18 works fine? Can I support this ticket to debug with someone of maintainers?

bersbersbers commented 4 years ago

In #26019 (which could be merged here as a duplicate), I describe two cases of this where the same video (the same bits) is provided through different formats. This may be relevant when one format fails - there may be another, equivalent format to use. This may imply that there may be cases where it may not even be needed to inform the user in case of fallback. Also, different formats might for used for continuation of each other. See #26019 for more details.

dyewts commented 3 years ago

Also having this issue with a few select videos. My -f argument chooses the best format codes, e.g. 248+251 for 1080p webm video and HQ webm audio but 248 gives a 404. Manually I can fix this by choosing 137+251 for 1080p mp4 video but a fallback function to automate this would be really helpful, especially because running the same list of channels multiple times to get new videos results in the manually downloaded videos being overwritten again.