Dailymotion download fails

Maxou44 commented 5 years ago

Checklist

[ X] I'm reporting a broken site support issue
[ X] I've verified that I'm running youtube-dl version 2019.04.30
[ X] I've checked that all provided URLs are alive and playable in a browser
[ X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ X] I've searched the bugtracker for similar bug reports including closed ones
[ X] I've read bugs section in FAQ

Verbose log

root# youtube-dl -v https://www.dailymotion.com/video/x77etcs
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.dailymotion.com/video/x77etcs']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.30
[debug] Python version 2.7.12 (CPython) - Linux-4.4.0-130-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 4.0.1-0york0, ffprobe 4.0.1-0york0
[debug] Proxy map: {}
[dailymotion] x77etcs: Downloading webpage
[dailymotion] x77etcs: Downloading metadata JSON
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[dailymotion] x77etcs: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://www.dailymotion.com/cdn/H264-1920x1080/video/x77etcs.mp4?sec=TLkyXUVASQiUwwiDAP3V2LCMl9jcAamoNbjegvoK00S2ysmYIO8nz9F337m8qfaD'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

Description

I can't download any Dailymotion video, I think the Dailymotion API/routes changed...

Urls to test: https://www.dailymotion.com/video/x77exhx https://www.dailymotion.com/video/x7706ll

dstftw commented 5 years ago

> py -3.7 .\youtube_dl\__main__.py https://www.dailymotion.com/video/x7706ll
[dailymotion] x7706ll: Downloading webpage
[dailymotion] x7706ll: Downloading metadata JSON
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[dailymotion] x7706ll: Downloading m3u8 information
[download] Destination: Les niches fiscales des entreprises dans le collimateur de Bercy-x7706ll.mp4
[download]   1.2% of 136.93MiB at 600.15KiB/s ETA 03:50
ERROR: Interrupted by user

Maxou44 commented 5 years ago

I have additionnal information, it looks like Dailymotion CDN banned ranges of server IP address, the solution is to use a proxy 🙂

RealDeviil commented 5 years ago

I am having the same issue:

root@waw:/tmp/youtube-dl# youtube-dl --verbose --write-pages http://www.dailymotion.com/video/x31j7ik_chapman-stick_music
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'--write-pages', u'http://www.dailymotion.com/video/x31j7ik_chapman-stick_music']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.05.11
[debug] Python version 2.7.9 (CPython) - Linux-4.14.63-xxxx-std-ipv6-64-x86_64-with-Debian-8
[debug] exe versions: ffmpeg N-87327-g18821e3ba1-static, ffprobe N-87327-g18821e3ba1-static
[debug] Proxy map: {}
[dailymotion] x31j7ik: Downloading webpage
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_video_x31j7ik.dump
[dailymotion] x31j7ik: Downloading metadata JSON
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_player_metadata_video_x31j7ikembedder=http%3A%2F%2Fwww.dailymotion.com%2Fvideo%2Fx31j7ik_chapman-stick_music_GK_PV5_NEON=1_integration=inline.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn_include=144.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn_include=380.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn_include=240.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn_include=720.dump
[dailymotion] x31j7ik: Downloading m3u8 information
[dailymotion] Saving request to x31j7ik_https_-_www.dailymotion.com_cdn_manifest_video_x31j7ik.m3u8sec=x2w05vBlNMAHkSG7f3Wf0Xb1ypGf9AvFxnqAqIOm7zGZ7EDWct9oiYqs8KFrcfsn_include=480.dump
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://www.dailymotion.com/cdn/H264-1280x720/video/x31j7ik.mp4?sec=mGzGq-Rt8hcoLMUUrI3ha7x2cNV_04RgYEAVOKXM7Dediqkp14TIIkU9Swx2S6mQ'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in process_info
    success = dl(filename, info_dict)
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "/usr/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/bin/youtube-dl/youtube_dl/downloader/http.py", line 341, in real_download
    establish_connection()
  File "/usr/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 469, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 656, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden

I'm running youtube-dl version 2019.05.11 on OVH dedicated servers in Poland and Canada.

I've tested these URLs: https://www.dailymotion.com/video/x1ces6k https://www.dailymotion.com/video/x31j7ik

I think this issue deserves more attention and testing. Even if Sergey cannot reproduce the issue, it is very possible that the issue affects a significant percentage of youtube-dl users.

Maxime, can you tell me more about your solution? What kind of proxy did you use to circumvent this issue? In what country is your proxy located?

Thank you.

Maxou44 commented 5 years ago

My server is located in France in an Online datacenter, I use a Surfshark proxy to solve this issue

flackune commented 5 years ago

I've got a Server in an OVH datacenter and another also in an Online datacenter. I opened a ticket to Dailymotion's support to find out more. I will keep you informed as soon as I have more details :)

RealDeviil commented 5 years ago

I guess if I have to use a proxy workaround, I will. But sometimes if you dig around in the source code, you can find a code workaround there as well. I would prefer a code workaround.

Using my own "test" code, I attempted to download videos from different sources, and pass different cookies and request headers, but nothing worked for me. :/

DeHackEd commented 5 years ago

I've experienced the same thing. Changing IP addresses isn't necessary though - downloading again using wget (use -v to make youtube-dl print the download URL it tries to fetch) has been successful for me.

mcmarkj commented 4 years ago

Can this be reopened? I'm having ongoing issues with this.

IRGC commented 4 years ago

@flackune any update bruh?

Karif commented 4 years ago

Problem continues...

youtube -v "https://www.dailymotion.com/video/x6uoseg"

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'https://www.dailymotion.com/video/x6uoseg'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2019.11.05 [debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-66-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6 [debug] Proxy map: {} [dailymotion] x6uoseg: Downloading webpage [dailymotion] x6uoseg: Downloading metadata JSON [dailymotion] x6uoseg: Downloading m3u8 information WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "/usr/local/bin/youtube/youtube_dl/YoutubeDL.py", line 796, in extract_info ie_result = ie.extract(url) File "/usr/local/bin/youtube/youtube_dl/extractor/common.py", line 530, in extract ie_result = self._real_extract(url) File "/usr/local/bin/youtube/youtube_dl/extractor/dailymotion.py", line 260, in _real_extract self._sort_formats(formats) File "/usr/local/bin/youtube/youtube_dl/extractor/common.py", line 1327, in _sort_formats raise ExtractorError('No video formats found') ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

remitamine commented 4 years ago

youtube-dl https://www.dailymotion.com/video/x6uoseg
[dailymotion] x6uoseg: Downloading webpage
[dailymotion] x6uoseg: Downloading metadata JSON
[dailymotion] x6uoseg: Downloading m3u8 information
[download] Destination: Tele1 TV - Canlı Yayın-x6uoseg.mp4
ffmpeg version n4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-VERSION:3')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:09.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:12.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:15.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:18.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:21.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:24.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:27.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:30.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:33.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:36.000+0100')
[hls @ 0x55b82e8fb340] Opening 'https://stream-05.dc3.dailymotion.com/sec(oGEyQzojvNQy_SUQu81YX_Y007_YwMWBJdzZruuf6M0)/dm/3/x6uoseg/s/stream-3-1573431808.ts' for reading
[hls @ 0x55b82e8fb340] Opening 'https://stream-05.dc3.dailymotion.com/sec(oGEyQzojvNQy_SUQu81YX_Y007_YwMWBJdzZruuf6M0)/dm/3/x6uoseg/s/stream-3-1573431811.ts' for reading
Input #0, hls, from 'https://stream-05.dc3.dailymotion.com/sec(oGEyQzojvNQy_SUQu81YX_Y007_YwMWBJdzZruuf6M0)/dm/3/x6uoseg/s/live-3.m3u8':
  Duration: N/A, start: 1620.896322, bitrate: N/A
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Tele1 TV - Canlı Yayın-x6uoseg.mp4.part':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 25 fps, 25 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=   41 fps=0.0 q=-1.0 size=     256kB time=00:00:01.69 bitrate=1237.4kbits/frame=   52 fps= 47 q=-1.0 size=     512kB time=00:00:02.08 bitrate=2007.2kbits/[hls @ 0x55b82e8fb340] Skip ('#EXT-X-VERSION:3')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:12.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:15.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:18.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:21.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:24.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:27.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:30.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:33.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:36.000+0100')
[hls @ 0x55b82e8fb340] Skip ('#EXT-X-PROGRAM-DATE-TIME:2019-11-11T01:23:39.000+0100')
[https @ 0x55b82ef39e00] Opening 'https://stream-05.dc3.dailymotion.com/sec(oGEyQzojvNQy_SUQu81YX_Y007_YwMWBJdzZruuf6M0)/dm/3/x6uoseg/s/stream-3-1573431814.ts' for reading
frame=   74 fps= 32 q=-1.0 size=     512kB time=00:00:02.99 bitrate=1400.4kbits

ytdl-org / youtube-dl