Closed xtechgamer735 closed 1 year ago
As in #31363,
... I can't reproduce the 403, but maybe you displeased the ITV content servers somehow.
Maybe don't tell them about your cookies?
$ python -m youtube_dl -v --test 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--test', u'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f7e7fbc36
[debug] Python 2.7.18 (CPython i686 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial - OpenSSL 1.1.1t 7 Feb 2023 - glibc 2.15
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: hdntl=exp=1684703036~acl=%2f2-3168-0065-001%2f*~data=hdntl,nohubplus~hmac=5661b69c3d5ff98af1387a21dc46982c1cc066f15f32c3ed080a8a8a9d3c36f9; bm_sz=0645B7DA0C73B105D79BE726B7E0785D~YAAQDT4SAn+41TSIAQAA/eX5OhOhE1i3BwSXGMW1f2V+We9xKQ2Pak4GjFwm1u0kzsf9cjg28+GPUbfx4kMcM77KV9NA/D/HTtdZFRfRGu8oDrsgU0q9oD8OpnZbmUDoBiYwJGyYclQZSFAqpELhKp+0TA6KUN6M+XKWkm1giL+uXzaSl3+jznD7+PBfkdkVy8wbxTdydm7faeES3o6rV0KC2kGwo++XLAT1ah0UzoeJ0vAxNcHIWF4OloQX82TIiNq5ghkVbYzjzILAMQ92XKB6qazd81ySZk6YKI4KML8=~4338758~3158833; _abck=D4FA764DB48004CED8ED016DA6F0F4C1~-1~YAAQDT4SAn641TSIAQAA/eX5Oglsgv+PA5Q6Z04N1A7NAn+4a03TOAkWo7wQIP5cHEvRyl/cZJH89sZWyWxcPPdQ2RWDBvbBaRO6l2VraoXItyLHZh1P7fbmoy8WhRkFKIdX4d7y0USsqAZsk3mYw97ZEhuoU2PS6nseguml9VV9eYIB8l/Hn4xnRcKG6CNYpp0sMDY6WSQTgz6rIVbG19aMtyO6K3H/F3domrSbi1EGAAp+Iet85nbr0iO7lhei946fC6kz0gS69UztVuS5UMSYDsOOnexRxz1U1ymvM9n3frBVRDzpd1FmrDrswwfr2dcdsnRm9+8VE7Ykb9F6r+KzqI4kIVGTFxEUdWu/oek/nKzHYsY=~-1~-1~-1
User-agent: Mozilla/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3706.3 Safari/537.36
' -i 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8' -c copy -fs 10241 -f mp4 'file:Heathrow - Britain'"'"'s Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part'
ffmpeg version 4.3-2ubuntu0~ppa16.04+8 Copyright (c) 2000-2020 the FFmpeg developers
...
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
[tcp @ 0x6edfc0] Starting connection attempt to 23.56.238.114 port 443
[tcp @ 0x6edfc0] Successfully connected to 23.56.238.114 port 443
[hls @ 0x6ea380] Skip ('#EXT-X-VERSION:4')
[hls @ 0x6ea380] Skip ('## Created with OEM (version=1.11.3-24483)')
[hls @ 0x6ea380] Skip ('#EXT-X-INDEPENDENT-SEGMENTS')
[hls @ 0x6ea380] Skip ('#USP-X-TIMESTAMP-MAP:MPEGTS=900000,LOCAL=1970-01-01T00:00:00Z')
[hls @ 0x6ea380] HLS request for url 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877-1.ts', offset 0, playlist 0
[hls @ 0x6ea380] Opening 'https://itvpnp.live.ott.irdeto.com/HLS/GetKey.aspx?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-3168-0065-001_6&KeyId=e45e2bd4-d6e7-43eb-ac4f-2e6558edbe17' for reading
[tcp @ 0x9c21c0] Starting connection attempt to 54.77.248.77 port 443
[tcp @ 0x9c21c0] Successfully connected to 54.77.248.77 port 443
[AVIOContext @ 0x6eec80] Statistics: 16 bytes read, 0 seeks
[hls @ 0x6ea380] Opening 'crypto+https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877-1.ts' for reading
[tcp @ 0x9c1dc0] Starting connection attempt to 23.56.238.128 port 443
[tcp @ 0x9c1dc0] Successfully connected to 23.56.238.128 port 443
[h264 @ 0xaaf480] Reinit context to 896x512, pix_fmt: yuvj420p
...
Input #0, hls, from 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8':
Duration: 00:46:26.16, start: 9.966667, bitrate: N/A
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
Metadata:
variant_bitrate : 0
Stream #0:1: Video: h264 (Main), 1 reference frame ([27][0][0][0] / 0x001B), yuvj420p(pc, bt709, left), 896x504 (896x512) [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
Metadata:
variant_bitrate : 0
Output #0, mp4, to 'file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part':
Metadata:
encoder : Lavf58.45.100
Stream #0:0: Video: h264 (Main), 1 reference frame (avc1 / 0x31637661), yuvj420p(pc, bt709, left), 896x504 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 25 fps, 25 tbr, 90k tbn, 90k tbc
Metadata:
variant_bitrate : 0
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
Metadata:
variant_bitrate : 0
Stream mapping:
Stream #0:1 -> #0:0 (copy)
Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
Automatically inserted bitstream filter 'aac_adtstoasc'; args=''
No more output streams to write to, finishing.
Not writing 'clli' atom. No content light level info.
Not writing 'mdcv' atom. Missing mastering metadata.
frame= 2 fps=0.0 q=-1.0 Lsize= 58kB time=00:00:00.07 bitrate=6459.9kbits/s speed= 229x
video:56kB audio:1kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.536401%
Input file #0 (https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8):
Input stream #0:0 (audio): 3 packets read (840 bytes);
Input stream #0:1 (video): 2 packets read (56919 bytes);
Total: 5 packets (57759 bytes) demuxed
Output file #0 (file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part):
Output stream #0:0 (video): 2 packets muxed (56919 bytes);
Output stream #0:1 (audio): 3 packets muxed (840 bytes);
Total: 5 packets (57759 bytes) muxed
[AVIOContext @ 0xab1ac0] Statistics: 2 seeks, 4 writeouts
[AVIOContext @ 0xa98600] Statistics: 200336 bytes read, 0 seeks
[AVIOContext @ 0x9559c0] Statistics: 37061 bytes read, 0 seeks
[ffmpeg] Downloaded 59224 bytes
[download] 100% of 57.84KiB in 00:02
$
Otherwise you might want to offer more details of your location, network setup, etc.
Thanks a lot for your response, I originally did this without cookies but was (and still am) getting the same result as with. I assumed this was to do with user auth. I have tried installing the latest commit with pip and achieved the same result as before, using your command above.
I am in the UK, I am not using a proxy/VPN and I am able to watch the programs in the browser fine using my account. I was able to reproduce the error above using my mobile hotspot too on Mac (both brew and python) and using a nightly .exe build on a windows machine. I hope this is enough info, please let me know if you need more.
Thanks a lot in advance,
There's a header capitalisation issue between Py2 (works) and Py 3 (fails):
--- old/youtube_dl/extractor/itv.py
+++ new/youtube_dl/extractor/itv.py
@@ -59,7 +59,7 @@ class ITVBaseIE(InfoExtractor):
@staticmethod
def _vanilla_ua_header():
- return {'User-agent': 'Mozilla/5.0'}
+ return {'User-Agent': 'Mozilla/5.0'}
def _download_webpage_handle(self, url, video_id, *args, **kwargs):
# specialised to (a) use vanilla UA (b) detect geo-block
Well spotted! Manually defining that header (--add-header User-Agent:Mozilla/5.0) is working perfectly for me now. Thanks for your help.
Using the latest daily/nightly Windows build, youtube-dl.exe
, which is built on py3.4, the download works (for me) if the default external downloader (ffmpeg
) is used for fragment download and decryption:
yt-dl -v --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[debug] ffmpeg command line: ".\FFmpeg\ffmpeg" -y -loglevel verbose -v 8 -stats -headers
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
User-agent: Mozilla/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3663.2 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Cookie: (removed for brevity)"
-i "https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8" -c copy -fs 10241 -f mp4 "-bsf:a" aac_adtstoasc "file:Heathrow - Britain'sBusiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part"
ffmpeg version n6.1-dev-705-N-110589-g30c71ef Copyright (c) 2000-2023 the FFmpeg developers
built with Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31944 for x86
configuration: (removed for brevity)
libavutil 58. 7.100 / 58. 7.100
libavcodec 60. 11.100 / 60. 11.100
libavformat 60. 5.100 / 60. 5.100
libavdevice 60. 2.100 / 60. 2.100
libavfilter 9. 8.100 / 9. 8.100
libswscale 7. 2.100 / 7. 2.100
libswresample 4. 11.100 / 4. 11.100
frame= 0 fps=0.0 q=-1.0 size= 0kB time=00:00:00.03 bitrate= 0.0kbits/
frame= 32 fps=0.0 q=-1.0 Lsize= 262kB time=00:00:01.51 bitrate=1420.5kbits/s speed= 9.7x
[ffmpeg] Downloaded 268706 bytes
[download] 100% of 262.41KiB in 00:03
The reason is found below:
User-agent: Mozilla/5.0
...
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3663.2 Safari/537.36
i.e. while itv.py
itself injects "User-agent", ffmpeg
in turn injects "User-Agent" π ...
OTOH, delegating HLSe download and decryption to [hlsnative] (implies built-in crypto
support) will fail π , as reported above:
yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--hls-prefer-native', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 1992, in process_info
File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 1931, in dl
File "D:\a\youtube-dl\youtube-dl\youtube_dl\downloader\common.py", line 376, in download
File "D:\a\youtube-dl\youtube-dl\youtube_dl\downloader\hls.py", line 64, in real_download
File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 2355, in urlopen
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 470, in open
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 580, in http_response
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 508, in error
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 442, in _call_chain
File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 588, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden
It has to be noted this is a recent-ish ITV change π that affects specifically the variant
playlists (but not the master
one), because I did use the "fixed" itvIE back in mid-February of this year (as bundled inside the py3.4 youtube-dl-exe
) to test-fetch some programmes with [hlsnative], and at the time "a" vs "A" wasn't an issue...
@dirkf , any plans to merge the patch to master
?
... With the patch applied and [hlsnative]:
yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--hls-prefer-native', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 465
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[download] 100% of 10.00KiB in 00:01
[debug] ffmpeg command line: ".\FFmpeg\ffprobe" -show_streams "file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4"
... Many thanks for your unwavering efforts! π
With my current dev version:
$ python3.5 -m pip list 2>&1 | fgrep crypto
cryptography 3.3.2
pycrypto 2.6.1
pycryptodomex 3.11.0
$ python3.5 -m youtube_dl -v --test 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065' --hls-prefer-native
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065', '--hls-prefer-native']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f55ab2716
[debug] Python 3.5.2 (CPython 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial (OpenSSL 1.0.2n 7 Dec 2017, glibc 2.9)
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 465
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[download] 100% of 10.00KiB in 00:00
[debug] ffmpeg command line: ffprobe -show_streams 'file:Heathrow - Britain'"'"'s Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4'
$
So clearly an update to master is needed.
There's a header capitalisation issue between Py2 (works) and Py 3 (fails):
In addition to my previous post above, I find above statement not to be entirely true, at least under Windows π (?) ...
Using my own compiled youtube-dl.exe
, built with CPython 2.7.18+PyCrypto-2.6.1, I find that the current itvIE
with "User-agent" doesn't work even in the case of Py2, when [hlsnative] is specified:
yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--ffmpeg-location', u'.\\FFmpeg', u'--external-downloader-args', u'-v 8 -stats', u'-v', u'--hls-prefer-native', u'--test', u'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.11
[debug] Lazy loading extractors enabled
[debug] Python 2.7.18 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2t 10 Sep 2019, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "youtube_dl\YoutubeDL.pyo", line 1992, in process_info
File "youtube_dl\YoutubeDL.pyo", line 1931, in dl
File "youtube_dl\downloader\common.pyo", line 376, in download
File "youtube_dl\downloader\hls.pyo", line 64, in real_download
File "youtube_dl\YoutubeDL.pyo", line 2355, in urlopen
File "urllib2.pyo", line 435, in open
File "urllib2.pyo", line 548, in http_response
File "urllib2.pyo", line 473, in error
File "urllib2.pyo", line 407, in _call_chain
File "urllib2.pyo", line 556, in http_error_default
HTTPError: HTTP Error 403: Forbidden
dirkf's log above demonstrates download success under py2.7 with ffmpeg
as the HLSe downloader...
OP (under MacOS, though) does use ffmpeg-6.0
as the default (external) downloader (py3.11); in his debug log however, I see that the ffmpeg
provided "User-Agent" is injected/sent first, whereas the itv.py
provided "User-agent" is injected/sent second; in the logs provided by dirkf (ffmpeg-4.3
, py2.7) and myself (ffmpeg-n6.1-dev
, py3.4), the opposite is true, so perhaps that's the deciding factor in this bug?
IMHO, this bug should be re-opened, until properly mitigated by a relevant PR or master
branch commit...
The dev version works equally with Py 2.7 where
$ python2.7 -m pip list 2>&1 | fgrep crypto
pycrypto 2.6.1
pycryptodomex 3.4.3
$
The patch at https://github.com/ytdl-org/youtube-dl/issues/32217#issuecomment-1556264918 is not the only place where capitalisation went wrong.
Checklist
Question
This problem follows from #31363 as I am seeing the same thing as the final comment from the OP. I am unable to comment there however.
I am trying to download a program from ITV Hub, using the --HEAD version of YouTube-dl installed on MacOS via Home-brew. The command I am using is:
youtube-dl "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" --cookies /Users/me/Downloads/www.itv.com_cookies.txt
Whenever I attempt this, I get the following error:
me@My-MBP ~ % youtube-dl "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" --cookies /Users/me/Downloads/www.itv.com_cookies.txt [ITV] 2a3168a0065: Downloading webpage [ITV] 2a3168a0065: Downloading JSON metadata [ITV] 2a3168a0065: Downloading m3u8 information [download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4 ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers built with Apple clang version 14.0.0 (clang-1400.0.29.202) configuration: --prefix=/usr/local/Cellar/ffmpeg/6.0 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [https @ 0x7fafa5004540] HTTP error 403 Forbidden https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8: Server returned 403 Forbidden (access denied)
Verbose log available here: https://pastebin.com/MkuziuJX
I'm sure I've missed something, but any help is appreciated! The program above is just an example, this happens for any program regardless of machine or network. Thanks in advance.