ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.19k stars 9.93k forks source link

Unable to download from ITV Hub #30578

Closed jmoorey1 closed 1 year ago

jmoorey1 commented 2 years ago

Checklist

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.4.0-96-generic-x86_64-with-Ubuntu-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {'no': 'localhost,127.0.0.1,::1'}
[debug] Using fake IP 25.170.173.6 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
ERROR: Unable to download webpage: ('The read operation timed out',) (caused by SSLError('The read operation timed out',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2737, in https_open
    req, **kwargs)
  File "/usr/lib/python2.7/urllib2.py", line 1208, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1178, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 452, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 408, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 641, in read
    v = self._sslobj.read(len)

Description

Trying to download episodes of Celebrity Juice from the ITV Hub, but it's halting after 'Downloading webpage' and then several minutes later it errors out.

jmoorey1 commented 2 years ago

Tried several additional switches, but alas to no avail.

Including the verbose log from '--print-traffic' to hopefully shed a little more light;

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--no-check-certificate', '--print-traffic', '-C', '--no-geo-bypass', '--proxy', '', '--verbose', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-96-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {}
send: b'GET /ip HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Mon, 31 Jan 2022 10:02:15 GMT
header: Content-Type: text/plain;charset=UTF-8
header: Transfer-Encoding: chunked
header: Connection: close
header: Vary: Accept-Encoding
header: Server: nginx
header: X-Xss-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Referrer-Policy: strict-origin-when-cross-origin
header: Content-Encoding: gzip
[debug] Public IP address: XXX.XXX.XXX.XXX
send: b'GET /latest/version HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Mon, 31 Jan 2022 10:02:15 GMT
header: Content-Type: text/plain; charset=UTF-8
header: Content-Length: 10
header: Connection: close
header: Server: nginx
header: Last-Modified: Thu, 16 Dec 2021 19:01:37 GMT
header: ETag: "a-5d3480d45fe58"
header: Accept-Ranges: bytes
header: X-Xss-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Referrer-Policy: strict-origin-when-cross-origin
[ITV] 1a7045a0288: Downloading webpage
send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
ERROR: Unable to download webpage: The read operation timed out (caused by timeout('The read operation timed out'))
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/utils.py", line 2735, in https_open
    return self.do_open(functools.partial(
  File "/usr/lib/python3.8/urllib/request.py", line 1358, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
dirkf commented 2 years ago

The betting is that ITV has employed a bouncer that rejects or ignores your web requests if it doesn't like the look of them.

For the moment, --user-agent 'Mozilla/5.0' confuses the bouncer into allowing access, like some suede trainers.

In your log, yt-dl sent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36 as if it was Chrome 70 on Windows 10 64-bit, but perhaps the bouncer only accepts that UA with HTTP/2 (Chrome 51 on).

Vangelis66 commented 2 years ago

Nice find, indeed! šŸŽ‰ (but ITVHub people are known to monitor downloading apps šŸ˜  , so am wondering how long it'll be before it gets patched... ); additionally, OP is employing v2021.12.17 which, AFAICT, doesn't include #30266; perhaps the latter could be further "improved" by adding this latest "fix", what do you reckon? šŸ˜œ

Vangelis66 commented 2 years ago

... OTOH, yt-dlp appears unaffected by this latest ITVHub shenanigan, so "they" must have dealt with it, inadvertently in this case, somehow differently:

yt-dlp -v -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288" => 

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale cp1253, fs utf-8, out utf-8 (No ANSI), err utf-8 (No ANSI), pref cp1253
[debug] yt-dlp version 2022.01.31 [7ba35c8] (win_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python version 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] Using fake IP 25.155.40.159 (GB) as X-Forwarded-For
[debug] [ITV] Extracting URL: https://www.itv.com/hub/celebrity-juice/1a7045a0288
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext,aext, hasaud, source, id
[info] Available formats for 1a7045a0288:
ID       EXT RESOLUTION FPS |   FILESIZE   TBR PROTO  | VCODEC        VBR ACODEC     ABR
----------------------------------------------------------------------------------------
hls-136  mp4 audio only     | ~ 33.84MiB  136k m3u8_n | audio only        mp4a.40.2 128k
hls-318  mp4 512x288     50 | ~ 79.13MiB  318k m3u8_n | avc1.4D401E  171k mp4a.40.2 128k
hls-601  mp4 512x288     50 | ~149.56MiB  601k m3u8_n | avc1.4D401F  438k mp4a.40.2 128k
hls-802  mp4 512x288     50 | ~199.58MiB  802k m3u8_n | avc1.4D401F  628k mp4a.40.2 128k
hls-1082 mp4 896x504     50 | ~269.25MiB 1082k m3u8_n | avc1.4D401F  892k mp4a.40.2 128k
hls-1670 mp4 896x504     50 | ~415.58MiB 1670k m3u8_n | avc1.4D401F 1447k mp4a.40.2 128k
dirkf commented 2 years ago

yt-dlp is now wearing various shades of Chrome/90+ and sending some other slightly different HTTP headers. Otherwise, the yt_dlp extractor and PR #30266 should be doing the same thing.

Probably we should just use s/t like the Chrome/iOS UA which also satisfies TikTok (today).

Vangelis66 commented 2 years ago

various shades of Chrome/90+ and sending some other slightly different HTTP headers.

Well, ITVHub's "bouncer" doesn't appear to only check for palatable UA strings; I changed my browser's default UA to Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36, which is the one rejected in OP's second log, and my browser has no issue loading that programme's page (cache and cookies fully wiped out before the page load):

itvhub

dirkf commented 2 years ago

I expect your current desktop browser is sending the slightly different HTTP headers.

Vangelis66 commented 2 years ago

I performed some additional tests with youtube-dl.exe and it looks like Chrome/89 is the minimum Chrome version (at this time šŸ˜‰ ...) that www.itv.com will sanction when requested by yt-dl:

youtube-dl --user-agent "Chrome/88" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

exhibits the issue reported by OP, while

youtube-dl --user-agent "Chrome/89" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

successfully gets past the [ITV] 1a7045a0288: Downloading webpage stage and on to [ITV] 1a7045a0288: Downloading JSON metadata; actually, setting the Chrome version to a big, future, value also does the trick (e.g. --user-agent "Chrome/200"); JFYI... šŸ˜ƒ

boxgr0ve commented 2 years ago

Apologies but I don't fully understand the above - should this be working now using --user-agent "Chrome/89" ?

When using the example above, I get ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);

I am running youtube-dl version 2021.12.17

jmoorey1 commented 2 years ago

Thank you for the troubleshooting, I have altered the user-agent switch and I'm now getting passed the [ITV] 1a7045a0288: Downloading webpage stage.

I'm now having the same issue as @boxgr0ve - it errors out while trying to download the JSON - ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);

Full output here. Tried it with ITV credentials and without and neither work.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--print-traffic', '--user-agent', 'Chrome/90', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {'no': 'localhost,127.0.0.1,::1'}
[debug] Using fake IP 25.134.61.247 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nX-Forwarded-For: 25.134.61.247\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=UTF-8
header: grace: none
header: Server: nginx
header: Vary: Accept-Encoding
header: X-Cache-Hits: 0
header: X-Frame-Options: DENY
header: X-Akamai-Transformed: 9 19358 0 pmb=mTOE,1
header: Cache-Control: public, max-age=28, s-maxage=60
header: Expires: Wed, 09 Feb 2022 10:24:20 GMT
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Content-Length: 15851
header: Connection: close
header: X-Device-BrandName:
header: X-Device-Os:
header: X-Device-MobileBrowser:
header: X-Device-ModelName:
header: X-Device-Os-Version:
[ITV] 1a7045a0288: Downloading JSON metadata
send: b'POST /playlist/itvonline/ITV2/1_7045_0288.001 HTTP/1.1\r\nContent-Length: 374\r\nHost: magni.itv.com\r\nAccept: application/vnd.itv.vod.playlist.v2+json\r\nContent-Type: application/json\r\nHmac: E8EDCDB18AE0E032C1A1C8BDE616100F270993CA\r\nX-Forwarded-For: 25.134.61.247\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
send: b'{"user": {"itvUserId": "", "entitlements": [], "token": ""}, "device": {"manufacturer": "Safari", "model": "5", "os": {"name": "Windows NT", "version": "6.1", "type": "desktop"}}, "client": {"version": "4.1", "id": "browser"}, "variantAvailability": {"featureset": {"min": ["hls", "aes", "outband-webvtt"], "max": ["hls", "aes", "outband-webvtt"]}, "platformTag": "dotcom"}}'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Content-Type: application/json
header: Vary: Origin
header: Content-Length: 123
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Connection: close
header: Access-Control-Allow-Credentials: true
header: Access-Control-Allow-Origin: *
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
dirkf commented 2 years ago

Today, using #30266 and --user-agent 'Chrome/89':

$ python -m youtube_dl -F -v 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89' 
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288', u'--user-agent', u'Chrome/89']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 0945174fa
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 25.39.215.163 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[info] Available formats for 1a7045a0288:
format code  extension  resolution note
hls-136      mp4        audio only  136k , mp4a.40.2@128k
hls-318      mp4        512x288     318k , avc1.4D401E@ 171k, 50.0fps, mp4a.40.2@128k
hls-601      mp4        512x288     601k , avc1.4D401F@ 438k, 50.0fps, mp4a.40.2@128k
hls-802      mp4        512x288     802k , avc1.4D401F@ 628k, 50.0fps, mp4a.40.2@128k
hls-1082     mp4        896x504    1082k , avc1.4D401F@ 892k, 50.0fps, mp4a.40.2@128k
hls-1670     mp4        896x504    1670k , avc1.4D401F@1447k, 50.0fps, mp4a.40.2@128k (best)
$ python -m youtube_dl --test 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89' 
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 340
[download] Destination: Celebrity Juice - Series 25 - Episode 1-1a7045a0288.mp4
[download] 100% of 10.00KiB in 00:00
$

OT TV critic: see how I used the --test option to avoid getting a viewable version of the show itself! The selection of Plebs and Wossie as test shows in the extractor was more acceptable.

jmoorey1 commented 2 years ago

Perfect - installed that version of itv.py and it worked a treat - thank you

boxgr0ve commented 2 years ago

@jmoorey1 Apologies again - can you let me know how you downloaded itv.py and ran it? I've look all though https://github.com/ytdl-org/youtube-dl/pull/30266 and even downloaded yt-dlp but can't make sense of it. I hope you can appreciate this my first time trying to install a different version.

For reference I originally installed YouTube-dl on a Mac via home-brew (if it makes a difference).

Thanks in anticipation.

jmoorey1 commented 2 years ago

I'm not completely au fait with the filesystem of a Mac, but the file you're after can be found on the 'Files changed' tab of #30266 - here's a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py

You'll need to find out where the /youtube_dl/extractor/ folder is on your machine, and download the file overwriting the existing version. Probably best to take a copy of the original before you alter it. You'll probably be able to see the path somewhere in the output if you add the --verbose switch when you run the command.

Hope that helps.

Vangelis66 commented 2 years ago

@boxgr0ve wrote:

Apologies but I don't fully understand the above - should this be working now using --user-agent "Chrome/89"? ... I am running youtube-dl version 2021.12.17

... Apologies accepted šŸ˜„ , but to anyone else, I already wrote that:

additionally, OP is employing v2021.12.17 which, AFAICT, doesn't include https://github.com/ytdl-org/youtube-dl/pull/30266;

so the UA-hack implies first applying #30266 (as yet still unmerged in master...) šŸ˜‰

dirkf commented 2 years ago

The PR obviously needs an update to apply a different UA, either locally or for all extractors, before it can be merged. This isn't Microsoft or Google, you know.

Vangelis66 commented 2 years ago

This isn't Microsoft or Google, you know

... I never implied that šŸ˜œ (TBH, I've got a lot to blame both for... šŸ˜ž ) and your hard work is being highly appreciated šŸ‘ ; to re-iterate: current flavour of https://github.com/ytdl-org/youtube-dl/pull/30266 ("working" itv.py from there) has to be manually applied at this time, the procedure depends on how you installed/invoke yt-dl on your system; this is a prerequisite to issuing the --user-agent switch(es) mentioned in this thread for a successful ITVHub fetch...

Vangelis66 commented 2 years ago

here's a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py

... The link points to https://github.com/ytdl-org/youtube-dl/issues/url :wink: ; no doubt a mishap :smile: , but it should be fixed...

boxgr0ve commented 2 years ago

Thanks all, I finally managed to download something from ITV hub!! I probably made it a lot more complicated than it should have been but for anyone else reading this who's stupid like me, the file path where the new itv.py file needs to go (at least for me) is:

/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor

"After replacing the file, I did start getting an error: ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one."

So I installed ffmpeg and it all seemed to work as per the attached screenshot. When I've used YouTube-dl in the past I didn't normally get the yellow and purple text but hey it worked!! One thing I did notice was the quality wasn't particularly great, 896x504 however this is probably a limitation of ITV hub?

Screenshot 2022-02-10 at 18 09 57
dirkf commented 2 years ago

Previously you didn't have ffmpeg which is what is generating the diagnostics. Apparently the ffmpeg devs and whoever is responsible for ITV content playout disagree on the correct format of the stream.

You can use --external-downloader-args '-v 16' (or possibly, 8) to cut down the chatter from ffmpeg.

Vangelis66 commented 2 years ago

ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Have you tried the --hls-prefer-native switch? youtube-dl defaults to using ffmpeg for dumping AES-128 encrypted HLS streams (it's the type of mobile streams the ITV Info Extractor "harvests" in order to fetch the ITVHub programme :stuck_out_tongue_winking_eye: ), but said switch will dump the stream via a pure Python implementation, no external downloader needed... However, since this is on a Mac, you may have to install an additional "decrypting" python module (hlsnative dependency), such as the deprecated PyCrypro or the active PyCryprodome equivalent; consult some Mac documentation on how to install one of these modules... But, as your log shows, ffmpeg is still required to mux downloaded stream to the MP4 container, so it's a good thing you installed it anyway... :smiley:

... I did notice was the quality wasn't particularly great, 896x504 however this is probably a limitation of ITV hub?

Yes; look at the formats tables posted by me here and by dirkf here (resolution column) ...

Cheers :smile:

jmoorey1 commented 2 years ago

... The link points to https://github.com/ytdl-org/youtube-dl/issues/url šŸ˜‰ ; no doubt a mishap šŸ˜„ , but it should be fixed...

@Vangelis66 - fixed - nice spot ;-)