search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.7k stars 10.07k forks source link

Dailymotion abrupt 404 #29867

Closed attinderdhillon closed 3 years ago

attinderdhillon commented 3 years ago

Checklist

Verbose log

youtube-dl https://www.dailymotion.com/video/x2kxdqm -f best --verbose
[debug] System config: ['--prefer-free-formats']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.dailymotion.com/video/x2kxdqm', '-f', 'best', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.6 (CPython) - Linux-5.13.9-200.fc34.x86_64-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4
[debug] Proxy map: {}
[dailymotion] Downloading Access Token
[dailymotion] x2kxdqm: Downloading media JSON metadata
[dailymotion] x2kxdqm: Downloading metadata JSON
[dailymotion] x2kxdqm: Downloading m3u8 information
[debug] Invoking downloader on 'https://proxy-033.ix7.dailymotion.com/sec(TKCcVu7iYSGl26pWZ5hLrDOpeUuCwx0CZ-BO2JvAqk74w167ldzGxMNU5KuQtbdz71NLGoF_jr7APquz2lbrqg)/video/621/280/156082126_mp4_h264_aac_hq.mp4'
[download] Destination: GP1 Season Review 1990-001-x2kxdqm.mp4
[download]  38.0% of 315.47MiB at 125.17KiB/s ETA 26:39[download] Got server HTTP error: Downloaded 125829120 bytes, expected 330789408 bytes. Retrying (attempt 1 of 10)...
ERROR: unable to download video data: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1975, in process_info
    success = dl(filename, info_dict)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1914, in dl
    return fd.download(name, info)
  File "/usr/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python3.9/site-packages/youtube_dl/downloader/http.py", line 351, in real_download
    establish_connection()
  File "/usr/lib/python3.9/site-packages/youtube_dl/downloader/http.py", line 116, in establish_connection
    raise err
  File "/usr/lib/python3.9/site-packages/youtube_dl/downloader/http.py", line 110, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2287, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib64/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Description

After downloading 100-200MB it abruptly throws a 404 error , resuming doesn't work either ( 404 )

dirkf commented 3 years ago

This is a strange.

First of all, what happens is that the HTTP connection is reset ('server HTTP error'); the HTTP downloader tries to re-establish the download connection ('Retrying (attempt 1 of 10)'); the attempt fails with HTTP error 404, after which the downloader fails and does not try any of the promised 9 further attempts.

Further, after that, both the 'http-480-' formats repeatedly give an immediate 404 with yt-dl's downloader, but not with (eg) wget. As the same HTTP headers should be sent in either case, this is mysterious. However the wget download can also fail with 404.

A possible work-around is to select from the non-HTTP (in this case, HLS) downloads with -f best[format_id!^=http].

attinderdhillon commented 3 years ago

hls also failed

youtube-dl https://www.dailymotion.com/video/x2kxdqm -f hls-480-1 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.dailymotion.com/video/x2kxdqm', '-f', 'hls-480-1', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.04.07
[debug] Python version 3.9.5 (CPython) - Linux-5.11.0-31-generic-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.3.2-0, ffprobe 4.3.2-0, rtmpdump 2.4
[debug] Proxy map: {}
[dailymotion] Downloading Access Token
[dailymotion] x2kxdqm: Downloading media JSON metadata
[dailymotion] x2kxdqm: Downloading metadata JSON
[dailymotion] x2kxdqm: Downloading m3u8 information
[debug] Invoking downloader on 'https://proxy-16.sg1.dailymotion.com/sec(49Ubi9udHuLcl5WIYb0CSl5xgR3WKYoiwr2GCdWn4OAwdqJchZj5kUffotQR_kUU2q3vw5dw2uf1NSH6a4ST0Zr4x1LokjsytSMV3tjs7OQ)/video/621/280/156082126_mp4_h264_aac_hq.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 358
[download] Destination: GP1 Season Review 1990-001-x2kxdqm.mp4
[download]  36.9% of ~335.32MiB at 926.39KiB/s ETA 05:18ERROR: giving up after 10 retries
  File "/usr/bin/youtube-dl", line 33, in <module>
    sys.exit(load_entry_point('youtube-dl==2021.4.7', 'console_scripts', 'youtube-dl')())
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 479, in main
    _real_main(argv)
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 469, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 2059, in download
    res = self.extract_info(
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 799, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 838, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 872, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 1683, in process_video_result
    self.process_info(new_info)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 1967, in process_info
    success = dl(filename, info_dict)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 1906, in dl
    return fd.download(name, info)
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/hls.py", line 149, in real_download
    success, frag_content = self._download_fragment(
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/fragment.py", line 104, in _download_fragment
    success = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/http.py", line 363, in real_download
    self.report_error('giving up after %s retries' % retries)
  File "/usr/lib/python3/dist-packages/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 628, in report_error
    self.trouble(error_message, tb)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 590, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())
dirkf commented 3 years ago

The http-480 downloads seem to fail consistently at 125829120, and the HLS one above seems to be similar. Perhaps the DM player does some sort of handshake that the extractor isn't replicating.

Looks like this is the same issue as #28444, #26076, #26019 where it was found that the failing download didn't play in the browser.

Apparently the http-380 downloads can work:

$ youtube-dl -v --ignore-config -f http-380-1 'https://www.dailymotion.com/video/x2kxdqm'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', '-f', 'http-380-1', 'https://www.dailymotion.com/video/x2kxdqm']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.5.2 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[dailymotion] Downloading Access Token
[dailymotion] x2kxdqm: Downloading media JSON metadata
[dailymotion] x2kxdqm: Downloading metadata JSON
[dailymotion] x2kxdqm: Downloading m3u8 information
[debug] Invoking downloader on 'https://proxy-033.ix7.dailymotion.com/sec(BfxC_mFcxewjs1MfOuTaunO-qVVkhI25f92Syhdr4aNUlo6UNXSuoQDS3VokMe3JxocZL-gLNT0UKcMT06itnQ)/video/621/280/156082126_mp4_h264_aac.mp4'
[download] Destination: GP1 Season Review 1990-001-x2kxdqm.mp4
[download] 100% of 181.37MiB in 43:53
$
ghost commented 3 years ago

Is the 480p video actually playable to the end in the browser? I tried with Firefox and it stopped about 20+ minutes past and got this:

Unexpected error This video can’t be displayed right now, sorry Report ID: 1fe4t5qc71hpubnv3mb

and in the DevTools:

GET https://proxy-018.dc3.dailymotion.com/sec(49Ubi9udHuLcl5WIYb0CSl5xgR3WKYoiwr2GCdWn4OAwdqJchZj5kUffotQR_kUU2q3vw5dw2uf1NSH6a4ST0USTTEhyjrplvfuf0t-2Wyg)/frag(133)/video/621/280/156082126_mp4_h264_aac_hq.ts

503 Service Unavailable

attinderdhillon commented 3 years ago

I confirm 480p stopped working about 20+ minutes past. I checked some more videos I had errors that wouldn't play past xx minutes. So the issue is at Dailymotion. youtube-dl works fine when the video is playable till the end.