ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.01k stars 10.01k forks source link

[FoxNews] HTTP Error 403: Forbidden #17824

Open john-x opened 6 years ago

john-x commented 6 years ago

Please follow the guide below


Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.10.05. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?


The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []
[debug] User config: ['-f', '(bestvideo+bestaudio)[ext=webm][height <=? 720]/best[height=720]/(bestvideo+bestaudio)[height <=? 720]/best', '--prefer-free-formats', '--add-metadata', '-c', '--sub-lang', 'enUS', '--prefer-ffmpeg', '--cookies=/home/username/.config/cookies.txt', '--user-agent', 'Mozilla/5.0 (X11; Linux x86_64; rv:52.9) Gecko/20100101 Goanna/4.1 Firefox/52.9 PaleMoon/28.1.0']
[debug] Custom config: []
[debug] Command-line args: ['-s', 'https://video.foxnews.com/v/5614615980001/?#sp=watch-live', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.10.05
[debug] Python version 3.6.6 (CPython) - Linux-4.9.0-8-amd64-x86_64-with-debian-9.5
[debug] exe versions: ffmpeg 3.2.12-1, ffprobe 3.2.12-1, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[foxnews] Downloading Akamai AMP feed
[foxnews] 5614615980001: Downloading f4m manifest
WARNING: Unable to download f4m manifest: HTTP Error 403: Forbidden
[foxnews] 5614615980001: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 507, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/foxnews.py", line 73, in _real_extract
    'http://%s/v/feed/video/%s.js?template=fox' % (host, video_id))
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/amp.py", line 89, in _extract_feed_info
    self._sort_formats(formats)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1286, in _sort_formats
    raise ExtractorError('No video formats found')
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Streams under this site that require authentication return a HTTP Error 403: Forbidden (even when a Netscape cookie file is used after logging in through the browser).

john-x commented 6 years ago

That seems odd. The plugin is there and I'm able to open other streams of theirs that don't require authentication (like their live radio stream: https://video.foxnews.com/v/5599024308001). As long as I have the right cookies it should work, right?. Is there another bug report that references this issue? I searched before posting but didn't seem to find one.

If it's any help here's the print traffic from my previous attempt as well:

[foxnews] Downloading Akamai AMP feed
[foxnews] 5614615980001: Downloading f4m manifest
send: b'GET /v/feed/video/5614615980001.js?template=fox HTTP/1.1\r\nHost: video.foxnews.com\r\nCookie: AMCV_17FC406C5357BA6E0A490D4D%40AdobeOrg=2121618341%7CMCIDTS%7C17812%7CMCMID%7C17246021787248651375926418534040882674%7CMCAAMLH-1539092008%7C7%7CMCAAMB-1539495248%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1538897648s%7CNONE%7CMCAID%7CNONE; FXN_flk=1; FXN_v_idpp=https%3A%2F%2Fvideo.foxnews.com%2Fv%2F5614615980001; ajs_anonymous_id=%22ee3dfdba-07b0-482b-8b97-2d26c6cd3c2b%22; ajs_group_id=null; ajs_user_id=null; optimizelyEndUserId=oeu1538878511172r0.4282492071524938; s_pers=%20omtr_lv%3D1538890966636%7C1633498966636%3B%20omtr_lv_s%3DLess%2520than%25201%2520day%7C1538892766636%3B%20s_nr%3D1538890966650-Repeat%7C1541482966650%3B; seg_xid=b4cb5020-c818-4271-9126-da04521dc172; seg_xid_fd=sync.fxnetworks.com; seg_xid_ts=1537424468728; _cb=BNTofwDlbdYGC7b74U; _cb_ls=1; _chartbeat2=.1538878544616.1538890454248.1.DxxLA0R_LFXZFElxZVnE8DIqbgF.2; _v__chartbeat3=B-YGO3PAOzqDv3SHD; live_fn=https%3A//video.foxnews.com/v/5614615980001/%3F%23sp%3Dwatch-live\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.9) Gecko/20100101 Goanna/4.1 Firefox/52.9 PaleMoon/28.1.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server header: X-Powered-By header: Access-Control-Allow-Origin header: Content-Encoding header: Content-Length header: Content-Type header: Cache-Control header: Date header: Connection header: Vary WARNING: Unable to download f4m manifest: HTTP Error 403: Forbidden
[foxnews] 5614615980001: Downloading m3u8 information
send: b'GET /z/FNCGO_436151@436151/manifest.f4m?hdcore=3.4.0&plugin=aasp-3.4.0.132.124 HTTP/1.1\r\nHost: fncgohds-lh.akamaihd-staging.net\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.9) Gecko/20100101 Goanna/4.1 Firefox/52.9 PaleMoon/28.1.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server header: Mime-Version header: Content-Type header: Content-Length header: Expires header: Cache-Control header: Pragma header: Date header: Connection header: Set-Cookie header: X-Akamai-Staging WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
send: b'GET /hls/live/263399/FNCGOHLSv2/master.m3u8 HTTP/1.1\r\nHost: fncgohls-i.akamaihd.net\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.9) Gecko/20100101 Goanna/4.1 Firefox/52.9 PaleMoon/28.1.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Server header: Mime-Version header: Content-Length header: Expires header: Cache-Control header: Pragma header: Date header: Connection header: Access-Control-Allow-Headers header: Access-Control-Expose-Headers header: Access-Control-Allow-Methods header: Access-Control-Allow-Origin header: Content-Type 
harlanb commented 6 years ago

I started seeing this last year some time. Still happening: [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'--prefer-ffmpeg', u'-o', u'%(series)s - s%(season_number)02de%(episode_number)02d - %(title)s.%(ext)s', u'https://www.fox.com/watch/2e9b46c3bec1d71febd4ab82a929a37e/'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2018.10.05 [debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-32-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2 [debug] Proxy map: {} [FOX] 2e9b46c3bec1d71febd4ab82a929a37e: Downloading JSON metadata [FOX] 2e9b46c3bec1d71febd4ab82a929a37e: Downloading webpage [uplynk:preplay] 751cbef9e7a74795864f510a1459fb6a: Downloading JSON metadata [uplynk:preplay] 751cbef9e7a74795864f510a1459fb6a: Downloading m3u8 information [uplynk:preplay] 751cbef9e7a74795864f510a1459fb6a: Downloading JSON metadata [debug] Default format spec: bestvideo+bestaudio/best [debug] Invoking downloader on u'http://content-auso2.uplynk.com/751cbef9e7a74795864f510a1459fb6a/k.m3u8' [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 636 [download] Destination: The Gifted - s02e04 - outMatched.mp4 [download] 0.2% of ~1.65GiB at Unknown speed ETA Unknown ETAERROR: unable to download video data: HTTP Error 403: Forbidden Traceback (most recent call last): File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1908, in process_info success = dl(filename, info_dict) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1847, in dl return fd.download(name, info) File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download return self.real_download(filename, info_dict) File "/usr/local/bin/youtube-dl/youtube_dl/downloader/hls.py", line 168, in real_download self._prepare_url(info_dict, decrypt_info['URI'])).read() File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2211, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 403: Forbidden

[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'--prefer-ffmpeg', u'-o', u'%(series)s - s%(season_number)02de%(episode_number)02d - %(title)s.%(ext)s', u'https://www.fox.com/watch/a7980139ac6b1bed706dfa6abf8a2d0b/'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2018.10.05 [debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-32-generic-x86_64-with-Ubuntu-18.04-bionic [debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2 [debug] Proxy map: {} [FOX] a7980139ac6b1bed706dfa6abf8a2d0b: Downloading JSON metadata [FOX] a7980139ac6b1bed706dfa6abf8a2d0b: Downloading webpage [uplynk:preplay] cc4ee5577e85427bb8696d0e5e83b2fe: Downloading JSON metadata [uplynk:preplay] cc4ee5577e85427bb8696d0e5e83b2fe: Downloading m3u8 information [uplynk:preplay] cc4ee5577e85427bb8696d0e5e83b2fe: Downloading JSON metadata [debug] Default format spec: bestvideo+bestaudio/best [debug] Invoking downloader on u'http://content-ause5.uplynk.com/cc4ee5577e85427bb8696d0e5e83b2fe/k.m3u8' [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 650 [download] Destination: Lethal Weapon - s03e04 - Leo Getz Justice.mp4 [download] 0.2% of ~1.63GiB at Unknown speed ETA Unknown ETAERROR: unable to download video data: HTTP Error 403: Forbidden Traceback (most recent call last): File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1908, in process_info success = dl(filename, info_dict) File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1847, in dl return fd.download(name, info) File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download return self.real_download(filename, info_dict) File "/usr/local/bin/youtube-dl/youtube_dl/downloader/hls.py", line 168, in real_download self._prepare_url(info_dict, decrypt_info['URI'])).read() File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2211, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 403: Forbidden