Closed jmoorey1 closed 1 year ago
Tried several additional switches, but alas to no avail.
Including the verbose log from '--print-traffic' to hopefully shed a little more light;
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--no-check-certificate', '--print-traffic', '-C', '--no-geo-bypass', '--proxy', '', '--verbose', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-96-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {}
send: b'GET /ip HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Mon, 31 Jan 2022 10:02:15 GMT
header: Content-Type: text/plain;charset=UTF-8
header: Transfer-Encoding: chunked
header: Connection: close
header: Vary: Accept-Encoding
header: Server: nginx
header: X-Xss-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Referrer-Policy: strict-origin-when-cross-origin
header: Content-Encoding: gzip
[debug] Public IP address: XXX.XXX.XXX.XXX
send: b'GET /latest/version HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Mon, 31 Jan 2022 10:02:15 GMT
header: Content-Type: text/plain; charset=UTF-8
header: Content-Length: 10
header: Connection: close
header: Server: nginx
header: Last-Modified: Thu, 16 Dec 2021 19:01:37 GMT
header: ETag: "a-5d3480d45fe58"
header: Accept-Ranges: bytes
header: X-Xss-Protection: 1; mode=block
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: X-Content-Type-Options: nosniff
header: Referrer-Policy: strict-origin-when-cross-origin
[ITV] 1a7045a0288: Downloading webpage
send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
ERROR: Unable to download webpage: The read operation timed out (caused by timeout('The read operation timed out'))
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/utils.py", line 2735, in https_open
return self.do_open(functools.partial(
File "/usr/lib/python3.8/urllib/request.py", line 1358, in do_open
r = h.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
The betting is that ITV has employed a bouncer that rejects or ignores your web requests if it doesn't like the look of them.
For the moment, --user-agent 'Mozilla/5.0'
confuses the bouncer into allowing access, like some suede trainers.
In your log, yt-dl sent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36
as if it was Chrome 70 on Windows 10 64-bit, but perhaps the bouncer only accepts that UA with HTTP/2 (Chrome 51 on).
Nice find, indeed! š (but ITVHub people are known to monitor downloading apps š , so am wondering how long it'll be before it gets patched... ); additionally, OP is employing v2021.12.17
which, AFAICT, doesn't include #30266; perhaps the latter could be further "improved" by adding this latest "fix", what do you reckon? š
... OTOH, yt-dlp
appears unaffected by this latest ITVHub shenanigan, so "they" must have dealt with it, inadvertently in this case, somehow differently:
yt-dlp -v -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288" =>
[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale cp1253, fs utf-8, out utf-8 (No ANSI), err utf-8 (No ANSI), pref cp1253
[debug] yt-dlp version 2022.01.31 [7ba35c8] (win_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python version 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] Using fake IP 25.155.40.159 (GB) as X-Forwarded-For
[debug] [ITV] Extracting URL: https://www.itv.com/hub/celebrity-juice/1a7045a0288
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext,aext, hasaud, source, id
[info] Available formats for 1a7045a0288:
ID EXT RESOLUTION FPS | FILESIZE TBR PROTO | VCODEC VBR ACODEC ABR
----------------------------------------------------------------------------------------
hls-136 mp4 audio only | ~ 33.84MiB 136k m3u8_n | audio only mp4a.40.2 128k
hls-318 mp4 512x288 50 | ~ 79.13MiB 318k m3u8_n | avc1.4D401E 171k mp4a.40.2 128k
hls-601 mp4 512x288 50 | ~149.56MiB 601k m3u8_n | avc1.4D401F 438k mp4a.40.2 128k
hls-802 mp4 512x288 50 | ~199.58MiB 802k m3u8_n | avc1.4D401F 628k mp4a.40.2 128k
hls-1082 mp4 896x504 50 | ~269.25MiB 1082k m3u8_n | avc1.4D401F 892k mp4a.40.2 128k
hls-1670 mp4 896x504 50 | ~415.58MiB 1670k m3u8_n | avc1.4D401F 1447k mp4a.40.2 128k
yt-dlp is now wearing various shades of Chrome/90+ and sending some other slightly different HTTP headers. Otherwise, the yt_dlp extractor and PR #30266 should be doing the same thing.
Probably we should just use s/t like the Chrome/iOS UA which also satisfies TikTok (today).
various shades of Chrome/90+ and sending some other slightly different HTTP headers.
Well, ITVHub's "bouncer" doesn't appear to only check for palatable UA strings; I changed my browser's default UA to Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36
, which is the one rejected in OP's second log, and my browser has no issue loading that programme's page (cache and cookies fully wiped out before the page load):
I expect your current desktop browser is sending the slightly different HTTP headers.
I performed some additional tests with youtube-dl.exe
and it looks like Chrome/89 is the minimum Chrome version (at this time š ...) that www.itv.com
will sanction when requested by yt-dl
:
youtube-dl --user-agent "Chrome/88" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"
exhibits the issue reported by OP, while
youtube-dl --user-agent "Chrome/89" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"
successfully gets past the [ITV] 1a7045a0288: Downloading webpage
stage and on to [ITV] 1a7045a0288: Downloading JSON metadata
; actually, setting the Chrome version to a big, future, value also does the trick (e.g. --user-agent "Chrome/200"
); JFYI... š
Apologies but I don't fully understand the above - should this be working now using --user-agent "Chrome/89" ?
When using the example above, I get ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);
I am running youtube-dl version 2021.12.17
Thank you for the troubleshooting, I have altered the user-agent switch and I'm now getting passed the [ITV] 1a7045a0288: Downloading webpage
stage.
I'm now having the same issue as @boxgr0ve - it errors out while trying to download the JSON - ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);
Full output here. Tried it with ITV credentials and without and neither work.
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--print-traffic', '--user-agent', 'Chrome/90', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {'no': 'localhost,127.0.0.1,::1'}
[debug] Using fake IP 25.134.61.247 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nX-Forwarded-For: 25.134.61.247\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=UTF-8
header: grace: none
header: Server: nginx
header: Vary: Accept-Encoding
header: X-Cache-Hits: 0
header: X-Frame-Options: DENY
header: X-Akamai-Transformed: 9 19358 0 pmb=mTOE,1
header: Cache-Control: public, max-age=28, s-maxage=60
header: Expires: Wed, 09 Feb 2022 10:24:20 GMT
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Content-Length: 15851
header: Connection: close
header: X-Device-BrandName:
header: X-Device-Os:
header: X-Device-MobileBrowser:
header: X-Device-ModelName:
header: X-Device-Os-Version:
[ITV] 1a7045a0288: Downloading JSON metadata
send: b'POST /playlist/itvonline/ITV2/1_7045_0288.001 HTTP/1.1\r\nContent-Length: 374\r\nHost: magni.itv.com\r\nAccept: application/vnd.itv.vod.playlist.v2+json\r\nContent-Type: application/json\r\nHmac: E8EDCDB18AE0E032C1A1C8BDE616100F270993CA\r\nX-Forwarded-For: 25.134.61.247\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
send: b'{"user": {"itvUserId": "", "entitlements": [], "token": ""}, "device": {"manufacturer": "Safari", "model": "5", "os": {"name": "Windows NT", "version": "6.1", "type": "desktop"}}, "client": {"version": "4.1", "id": "browser"}, "variantAvailability": {"featureset": {"min": ["hls", "aes", "outband-webvtt"], "max": ["hls", "aes", "outband-webvtt"]}, "platformTag": "dotcom"}}'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Content-Type: application/json
header: Vary: Origin
header: Content-Length: 123
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Connection: close
header: Access-Control-Allow-Credentials: true
header: Access-Control-Allow-Origin: *
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/usr/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
Today, using #30266 and --user-agent 'Chrome/89'
:
$ python -m youtube_dl -F -v 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288', u'--user-agent', u'Chrome/89']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 0945174fa
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 25.39.215.163 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[info] Available formats for 1a7045a0288:
format code extension resolution note
hls-136 mp4 audio only 136k , mp4a.40.2@128k
hls-318 mp4 512x288 318k , avc1.4D401E@ 171k, 50.0fps, mp4a.40.2@128k
hls-601 mp4 512x288 601k , avc1.4D401F@ 438k, 50.0fps, mp4a.40.2@128k
hls-802 mp4 512x288 802k , avc1.4D401F@ 628k, 50.0fps, mp4a.40.2@128k
hls-1082 mp4 896x504 1082k , avc1.4D401F@ 892k, 50.0fps, mp4a.40.2@128k
hls-1670 mp4 896x504 1670k , avc1.4D401F@1447k, 50.0fps, mp4a.40.2@128k (best)
$ python -m youtube_dl --test 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89'
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 340
[download] Destination: Celebrity Juice - Series 25 - Episode 1-1a7045a0288.mp4
[download] 100% of 10.00KiB in 00:00
$
OT TV critic: see how I used the --test
option to avoid getting a viewable version of the show itself! The selection of Plebs and Wossie as test shows in the extractor was more acceptable.
Perfect - installed that version of itv.py and it worked a treat - thank you
@jmoorey1 Apologies again - can you let me know how you downloaded itv.py and ran it? I've look all though https://github.com/ytdl-org/youtube-dl/pull/30266 and even downloaded yt-dlp but can't make sense of it. I hope you can appreciate this my first time trying to install a different version.
For reference I originally installed YouTube-dl on a Mac via home-brew (if it makes a difference).
Thanks in anticipation.
I'm not completely au fait with the filesystem of a Mac, but the file you're after can be found on the 'Files changed' tab of #30266 - here's a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py
You'll need to find out where the /youtube_dl/extractor/
folder is on your machine, and download the file overwriting the existing version. Probably best to take a copy of the original before you alter it. You'll probably be able to see the path somewhere in the output if you add the --verbose
switch when you run the command.
Hope that helps.
@boxgr0ve wrote:
Apologies but I don't fully understand the above - should this be working now using --user-agent "Chrome/89"? ... I am running youtube-dl version 2021.12.17
... Apologies accepted š , but to anyone else, I already wrote that:
additionally, OP is employing v2021.12.17 which, AFAICT, doesn't include https://github.com/ytdl-org/youtube-dl/pull/30266;
so the UA-hack implies first applying #30266 (as yet still unmerged in master
...) š
The PR obviously needs an update to apply a different UA, either locally or for all extractors, before it can be merged. This isn't Microsoft or Google, you know.
This isn't Microsoft or Google, you know
... I never implied that š (TBH, I've got a lot to blame both for... š ) and your hard work is being highly appreciated š ; to re-iterate: current flavour of https://github.com/ytdl-org/youtube-dl/pull/30266 ("working" itv.py
from there) has to be manually applied at this time, the procedure depends on how you installed/invoke yt-dl
on your system; this is a prerequisite to issuing the --user-agent
switch(es) mentioned in this thread for a successful ITVHub fetch...
here's a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py
... The link points to https://github.com/ytdl-org/youtube-dl/issues/url
:wink: ; no doubt a mishap :smile: , but it should be fixed...
Thanks all, I finally managed to download something from ITV hub!! I probably made it a lot more complicated than it should have been but for anyone else reading this who's stupid like me, the file path where the new itv.py file needs to go (at least for me) is:
/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor
"After replacing the file, I did start getting an error: ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one."
So I installed ffmpeg and it all seemed to work as per the attached screenshot. When I've used YouTube-dl in the past I didn't normally get the yellow and purple text but hey it worked!! One thing I did notice was the quality wasn't particularly great, 896x504 however this is probably a limitation of ITV hub?
Previously you didn't have ffmpeg which is what is generating the diagnostics. Apparently the ffmpeg devs and whoever is responsible for ITV content playout disagree on the correct format of the stream.
You can use --external-downloader-args '-v 16'
(or possibly, 8) to cut down the chatter from ffmpeg.
ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.
Have you tried the --hls-prefer-native
switch? youtube-dl
defaults to using ffmpeg
for dumping AES-128 encrypted HLS streams (it's the type of mobile streams the ITV Info Extractor "harvests" in order to fetch the ITVHub programme :stuck_out_tongue_winking_eye: ), but said switch will dump the stream via a pure Python implementation, no external downloader needed...
However, since this is on a Mac, you may have to install an additional "decrypting" python module (hlsnative dependency), such as the deprecated PyCrypro or the active PyCryprodome equivalent; consult some Mac documentation on how to install one of these modules...
But, as your log shows, ffmpeg
is still required to mux downloaded stream to the MP4 container, so it's a good thing you installed it anyway... :smiley:
... I did notice was the quality wasn't particularly great, 896x504 however this is probably a limitation of ITV hub?
Yes; look at the formats tables posted by me here and by dirkf here (resolution column) ...
Cheers :smile:
... The link points to
https://github.com/ytdl-org/youtube-dl/issues/url
š ; no doubt a mishap š , but it should be fixed...
@Vangelis66 - fixed - nice spot ;-)
Checklist
Verbose log
Description
Trying to download episodes of Celebrity Juice from the ITV Hub, but it's halting after 'Downloading webpage' and then several minutes later it errors out.