fox.com needs fixing please.

[rellik31]

hi, can you please update youtube-dl so it works with the new fox.com urls

thank you

free https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/

login https://www.fox.com/watch/30056b295fb57f7452aeeb4920bc3024/

i use a youtube-dl gui so i don"t get any error messages but fox was working perfect a few days ago and could download locked videos that needed a login without logging in with no problems but now the fox site has changed the urls and site layout

[arisboch]

I fed the free url to youtube-dl, here's the output:

casval@casval-X6815 /tmp % youtube-dl -v https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.fox.com/watch/087036ca7f33c8eb79b08152b4dd75c1/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.02
[debug] Python version 2.7.13 - Linux-4.10.0-33-generic-x86_64-with-Ubuntu-17.04-zesty
[debug] exe versions: avconv 3.2.4-1build2, avprobe 3.2.4-1build2, ffmpeg 3.2.4-1build2, ffprobe 3.2.4-1build2, rtmpdump 2.4
[debug] Proxy map: {}
[FOX] 087036: Downloading webpage
ERROR: Unable to extract drupal settings; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 776, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 434, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/fox.py", line 35, in _real_extract
    webpage, 'drupal settings'), video_id)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 797, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract drupal settings; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

I have no fox.com password, so someone else do that.

[timendum]

It's also geo-restricted.

[arisboch]

@timendum Would it help if I tried to let it run again via Tor Onion with an US exit node?

[gkoelln]

I'm looking into it. It looks like a site redesign.

[CyberbobGR]

I can confirm this was working a few days ago. Now it shows this:

ERROR: Unable to extract drupal settings; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[phinneus1]

still major issues on there end.....I managed to feed m3u8 into youtube-dl on recent show.......720p dimensions where Width : 404 pixels Height : 228 pixels as was 432p and 1080p(1080 is new btw) funny looking at 6mb/s 228p video lol also ads(national ads only) are in video feed too atm........so lot of kinks on there end also

[rellik31]

can it be fixed in the next few days? fox are due to remove alot of episodes soon

thanks.

[phinneus1]

downlaoder works.........but all there stuff has embedded ads now :(

Note: runtime is correct on the website player but downaloded file is longer then that runtime......also the ads are a diff resolution/framerate or something cause it crashes avidemux when I scan threw them

[Nii-90]

Perhaps the easiest way to 'fix' the ad issue this would be for youtube-dl to start a new file every time UPLYNK-SEGMENT items occur in the m3u8 file, since UPLYNK-SEGMENT occurs at changes in the stream. Or alternately, to add an option to process each fragment like an individual video when --keep-video --keep-fragments is used. As it is right now, the Frag* files aren't playable. If they were, users could identify/delete the problematic fragments and stitch the items back together themselves.

[gkoelln]

Perhaps there's a way to filter out the "#UPLYNK-SEGMENT: ... , ... , ad" segments. (I'm assuming this would require a modification to common.py)

[Nii-90]

From what I encountered when dealing with The Orville, technically yes, but it comes with a couple significant caveats or drawbacks. Some are easy to filter out based on the video properties, but others could only rely on length and could potentially cause the end credits to be skipped too. I guess one way to know is if multiple streams occur under one UPLYNK-SEGMENT, it's probably part of the main program, but without checks on properties or length of the whole UPLYNK-SEGMENT you can't be sure (and it could always get trickier if all the ads suddenly start having the same properties as the main program*, or if the main program's UPLYNK-SEGMENTS suddenly start occurring mid-act; that's why I suggested having a playable-fragment fallback option so end users could brute force it themselves later).

*especially if this means they stop flagging the ads separately as well.

Comedy Central's extractor (despite probably being served in an entirely different fashion) puts each main segment of the program into a separate file, and always has for as far back as I can remember. A fringe bonus of that approach is that the end user can append all of the good segments back together with, say, MKVToolNix, and have it automatically insert chapter markers where the commercial breaks had been, since those are the natural places where chapters tend to be inserted anyway. AFAIK, chapter information is something youtube-dl has absolutely no control over (and would definitely be lost in some sort of smart filtering to a single output file), but outputting by big, main segments would allow the user to reconstruct that information with virtually no overhead or significant hassle.

[rpvcg]

The initial theplatform request should redirect to a preplay json.

The data contents of the preplay json should have a "playURL" (including pbs code) at the very end of the json data. That playURL can be parsed down to an ad-free parent hls master url.

For example, for a playURL that looks like this; https://content-ause2.uplynk.com/preplay2/6c68ec3ffc3c4df7972591ba15629fcf/447ef3b45492a2e10d37d1742781b57f/6luJp5COKhZUvGh2fSdjTB34T0DrGRd0si8T4ohVViFH.m3u8?pbs=b88ec83f195142119194f92cbd3eec47

Parse it to; https://content-ause2.uplynk.com/6c68ec3ffc3c4df7972591ba15629fcf.m3u8?pbs=b88ec83f195142119194f92cbd3eec47

[remitamine]

the method for bypassing Uplynk preplay ads were already been used in viceland extractor, it should be usable as well for fox, the uplynk preplay json url should be delegated to UplynkPreplayIE.

[mvbattista]

@remitamine For clarification (and since there has been any update on this), does that mean that all that needs to be done is to add 'add_ie': ['UplynkPreplay'], to the FOXIE class under _TESTS?

[remitamine]

does that mean that all that needs to be done is to add 'add_ie': ['UplynkPreplay'], to the FOXIE class under _TESTS?

no, the changes that need to be done is:

• do no extract the formats in https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/fox.py#L57-L63.
• extract the real uplynk preplay link(theplatform redirect to uplynk so it should be possible to get the url with a HEAD request).
• pass uplynk preplay url in the returned info dict(and also add 'ie_key': 'UplynkPreplay').

this is one way to do it. the other possibility is to make theplatform handle uplynk preplay urls and than delegate the extraction to ThePlatformIE.

[mvbattista]

@dstftw As you implemented the original extraction of the formats, can you please implement the fixes @remitamine suggested?

[mvbattista]

This commit does not handle videos behind a TV provider (e.g. some shows up to a week after original airing):

https://www.fox.com/watch/818529c3d72584486365f78ae851fa67/

[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--ap-mso', 'Verizon', '--ap-username', 'PRIVATE', 'https://www.fox.com/watch/818529c3d72584486365f78ae851fa67/']
Type TV provider account password and press [Return]:
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.09.15
[debug] Python version 3.6.0 - Darwin-17.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.3.4, ffprobe 3.3.4, rtmpdump 2.4
[debug] Proxy map: {}
[FOX] 818529c3d72584486365f78ae851fa67: Downloading JSON metadata
[FOX] 818529c3d72584486365f78ae851fa67: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 503, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2177, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)```

[jaylweb]

I concur. The commit does not handle videos behind a TV provider. Fox requires shows to be verified with a TV provider login the first week when posted online and then after 5 episodes are online.

https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/

[jaylweb]

The error would be helpful.

Download Video Streams
Enter video source link: https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--get-filename', '--ap-mso', 'DTV', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE',
'--rm-cache-dir', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrom
e/63.0.3218.0 Safari/537.36', '--verbose', '-f', 'mp4[height=720]', '-o', '%(series)s - s%(season_number)02de%(episode_nu
mber)02d.%(ext)s', 'https://www.fox.com/watch/2ae78a0e3d34f92628300a8ad460b3ec/']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2017.10.15.1
[debug] Python version 3.4.4 - Windows-10-10.0.15063
[debug] exe versions: ffmpeg N-82092-g89ec4ad, ffprobe N-82092-g89ec4ad, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https:/
/yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl w
ith the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplnz97faa\build\youtube_dl\extractor\common.py", line
 503, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmplnz97faa\build\youtube_dl\YoutubeDL.py", line 2195,
in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

[rellik31]

i also get the same error is it possible to fix?

[jaylweb]

@dstftw or @remitamine what you say?

I can test. Just let me know. :)

[rellik31]

jaylweb, have you found a fix yet? the old version oct 1st never gives a issue downloading from fox but it also downloads add videos (about 8mins worth)

i use the same dns number so i know the issue is the the new oct 29th version of yt-dl

i am new to downloading streams so i have no idea how to fix problems like this

[alyssarosenzweig]

diff --git a/youtube_dl/extractor/fox.py b/youtube_dl/extractor/fox.py
index 5f98d017b..75322f93e 100644
--- a/youtube_dl/extractor/fox.py
+++ b/youtube_dl/extractor/fox.py
@@ -98,7 +98,7 @@ class FOXIE(AdobePassIE):
         urlh = self._request_webpage(HEADRequest(release_url), video_id)
         video_url = compat_str(urlh.geturl())

-        if UplynkPreplayIE.suitable(video_url):
+        if False and UplynkPreplayIE.suitable(video_url):
             info.update({
                 '_type': 'url_transparent',
                 'url': video_url,

fixes it (for non-AP-locked videos)

[alyssarosenzweig]

And for Adobe Pass locked stuff. (Yes, I'm aware this patchset is awful, but I'm very new to the ytdl codebase, and I don't have the time to contribute. But I fixed this up and figured I should push back the changes, in case someone more knowledgable and patient can get it merged.)

diff --git a/youtube_dl/extractor/fox.py b/youtube_dl/extractor/fox.py
index 75322f93e..5a0ba85ea 100644
--- a/youtube_dl/extractor/fox.py
+++ b/youtube_dl/extractor/fox.py
@@ -1,6 +1,8 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import urllib
+
 from .adobepass import AdobePassIE
 from .uplynk import UplynkPreplayIE
 from ..compat import compat_str
@@ -47,6 +49,11 @@ class FOXIE(AdobePassIE):
     def _real_extract(self, url):
         video_id = self._match_id(url)

+        resource = self._get_mvpd_resource(
+            'fbc-fox', "null", video_id, "null") 
+        query_auth = self._extract_mvpd_auth(
+            url, video_id, 'fbc-fox', resource)
+
         video = self._download_json(
             'https://api.fox.com/fbc-content/v1_4/video/%s' % video_id,
             video_id, headers={
@@ -58,6 +65,9 @@ class FOXIE(AdobePassIE):
         title = video['name']
         release_url = video['videoRelease']['url']

+        if query_auth:
+            release_url = release_url + "&auth=" + urllib.quote(query_auth)
+
         description = video.get('description')
         duration = int_or_none(video.get('durationInSeconds')) or int_or_none(
             video.get('duration')) or parse_duration(video.get('duration'))
@@ -76,10 +86,6 @@ class FOXIE(AdobePassIE):
         episode_number = int_or_none(video.get('episodeNumber'))
         release_year = int_or_none(video.get('releaseYear'))

-        if data.get('authRequired'):
-            # TODO: AP
-            pass
-
         info = {
             'id': video_id,
             'title': title,

I also needed https://github.com/rg3/youtube-dl/pull/11258 with mekantor's fix in https://github.com/rg3/youtube-dl/issues/11257 and change the severity of raise_mvpd_required so it wouldn't die for automatic login.

[alyssarosenzweig]

...apparently above patchset is a duplicate of https://github.com/rg3/youtube-dl/pull/14489/files. RIP.