ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.36k stars 10.04k forks source link

ITV m3u8 403 Error #32217

Closed xtechgamer735 closed 1 year ago

xtechgamer735 commented 1 year ago

Checklist

Question

This problem follows from #31363 as I am seeing the same thing as the final comment from the OP. I am unable to comment there however.

I am trying to download a program from ITV Hub, using the --HEAD version of YouTube-dl installed on MacOS via Home-brew. The command I am using is: youtube-dl "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" --cookies /Users/me/Downloads/www.itv.com_cookies.txt

Whenever I attempt this, I get the following error:

me@My-MBP ~ % youtube-dl "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" --cookies /Users/me/Downloads/www.itv.com_cookies.txt [ITV] 2a3168a0065: Downloading webpage [ITV] 2a3168a0065: Downloading JSON metadata [ITV] 2a3168a0065: Downloading m3u8 information [download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4 ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers built with Apple clang version 14.0.0 (clang-1400.0.29.202) configuration: --prefix=/usr/local/Cellar/ffmpeg/6.0 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [https @ 0x7fafa5004540] HTTP error 403 Forbidden https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8: Server returned 403 Forbidden (access denied)

Verbose log available here: https://pastebin.com/MkuziuJX

I'm sure I've missed something, but any help is appreciated! The program above is just an example, this happens for any program regardless of machine or network. Thanks in advance.

dirkf commented 1 year ago

As in #31363,

... I can't reproduce the 403, but maybe you displeased the ITV content servers somehow.

Maybe don't tell them about your cookies?

$ python -m youtube_dl -v --test 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--test', u'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f7e7fbc36
[debug] Python 2.7.18 (CPython i686 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial - OpenSSL 1.1.1t  7 Feb 2023 - glibc 2.15
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: hdntl=exp=1684703036~acl=%2f2-3168-0065-001%2f*~data=hdntl,nohubplus~hmac=5661b69c3d5ff98af1387a21dc46982c1cc066f15f32c3ed080a8a8a9d3c36f9; bm_sz=0645B7DA0C73B105D79BE726B7E0785D~YAAQDT4SAn+41TSIAQAA/eX5OhOhE1i3BwSXGMW1f2V+We9xKQ2Pak4GjFwm1u0kzsf9cjg28+GPUbfx4kMcM77KV9NA/D/HTtdZFRfRGu8oDrsgU0q9oD8OpnZbmUDoBiYwJGyYclQZSFAqpELhKp+0TA6KUN6M+XKWkm1giL+uXzaSl3+jznD7+PBfkdkVy8wbxTdydm7faeES3o6rV0KC2kGwo++XLAT1ah0UzoeJ0vAxNcHIWF4OloQX82TIiNq5ghkVbYzjzILAMQ92XKB6qazd81ySZk6YKI4KML8=~4338758~3158833; _abck=D4FA764DB48004CED8ED016DA6F0F4C1~-1~YAAQDT4SAn641TSIAQAA/eX5Oglsgv+PA5Q6Z04N1A7NAn+4a03TOAkWo7wQIP5cHEvRyl/cZJH89sZWyWxcPPdQ2RWDBvbBaRO6l2VraoXItyLHZh1P7fbmoy8WhRkFKIdX4d7y0USsqAZsk3mYw97ZEhuoU2PS6nseguml9VV9eYIB8l/Hn4xnRcKG6CNYpp0sMDY6WSQTgz6rIVbG19aMtyO6K3H/F3domrSbi1EGAAp+Iet85nbr0iO7lhei946fC6kz0gS69UztVuS5UMSYDsOOnexRxz1U1ymvM9n3frBVRDzpd1FmrDrswwfr2dcdsnRm9+8VE7Ykb9F6r+KzqI4kIVGTFxEUdWu/oek/nKzHYsY=~-1~-1~-1
User-agent: Mozilla/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3706.3 Safari/537.36
' -i 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8' -c copy -fs 10241 -f mp4 'file:Heathrow - Britain'"'"'s Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part'
ffmpeg version 4.3-2ubuntu0~ppa16.04+8 Copyright (c) 2000-2020 the FFmpeg developers
...
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[tcp @ 0x6edfc0] Starting connection attempt to 23.56.238.114 port 443
[tcp @ 0x6edfc0] Successfully connected to 23.56.238.114 port 443
[hls @ 0x6ea380] Skip ('#EXT-X-VERSION:4')
[hls @ 0x6ea380] Skip ('## Created with OEM  (version=1.11.3-24483)')
[hls @ 0x6ea380] Skip ('#EXT-X-INDEPENDENT-SEGMENTS')
[hls @ 0x6ea380] Skip ('#USP-X-TIMESTAMP-MAP:MPEGTS=900000,LOCAL=1970-01-01T00:00:00Z')
[hls @ 0x6ea380] HLS request for url 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877-1.ts', offset 0, playlist 0
[hls @ 0x6ea380] Opening 'https://itvpnp.live.ott.irdeto.com/HLS/GetKey.aspx?CrmId=itvpnp&AccountId=itvpnp&ContentId=2-3168-0065-001_6&KeyId=e45e2bd4-d6e7-43eb-ac4f-2e6558edbe17' for reading
[tcp @ 0x9c21c0] Starting connection attempt to 54.77.248.77 port 443
[tcp @ 0x9c21c0] Successfully connected to 54.77.248.77 port 443
[AVIOContext @ 0x6eec80] Statistics: 16 bytes read, 0 seeks
[hls @ 0x6ea380] Opening 'crypto+https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877-1.ts' for reading
[tcp @ 0x9c1dc0] Starting connection attempt to 23.56.238.128 port 443
[tcp @ 0x9c1dc0] Successfully connected to 23.56.238.128 port 443
[h264 @ 0xaaf480] Reinit context to 896x512, pix_fmt: yuvj420p
...
Input #0, hls, from 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8':
  Duration: 00:46:26.16, start: 9.966667, bitrate: N/A
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Video: h264 (Main), 1 reference frame ([27][0][0][0] / 0x001B), yuvj420p(pc, bt709, left), 896x504 (896x512) [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0: Video: h264 (Main), 1 reference frame (avc1 / 0x31637661), yuvj420p(pc, bt709, left), 896x504 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 25 fps, 25 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
Automatically inserted bitstream filter 'aac_adtstoasc'; args=''
No more output streams to write to, finishing.
Not writing 'clli' atom. No content light level info.
Not writing 'mdcv' atom. Missing mastering metadata.
frame=    2 fps=0.0 q=-1.0 Lsize=      58kB time=00:00:00.07 bitrate=6459.9kbits/s speed= 229x    
video:56kB audio:1kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.536401%
Input file #0 (https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8):
  Input stream #0:0 (audio): 3 packets read (840 bytes); 
  Input stream #0:1 (video): 2 packets read (56919 bytes); 
  Total: 5 packets (57759 bytes) demuxed
Output file #0 (file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part):
  Output stream #0:0 (video): 2 packets muxed (56919 bytes); 
  Output stream #0:1 (audio): 3 packets muxed (840 bytes); 
  Total: 5 packets (57759 bytes) muxed
[AVIOContext @ 0xab1ac0] Statistics: 2 seeks, 4 writeouts
[AVIOContext @ 0xa98600] Statistics: 200336 bytes read, 0 seeks
[AVIOContext @ 0x9559c0] Statistics: 37061 bytes read, 0 seeks
[ffmpeg] Downloaded 59224 bytes
[download] 100% of 57.84KiB in 00:02
$

Otherwise you might want to offer more details of your location, network setup, etc.

xtechgamer735 commented 1 year ago

Thanks a lot for your response, I originally did this without cookies but was (and still am) getting the same result as with. I assumed this was to do with user auth. I have tried installing the latest commit with pip and achieved the same result as before, using your command above.

I am in the UK, I am not using a proxy/VPN and I am able to watch the programs in the browser fine using my account. I was able to reproduce the error above using my mobile hotspot too on Mac (both brew and python) and using a nightly .exe build on a windows machine. I hope this is enough info, please let me know if you need more.

Thanks a lot in advance,

dirkf commented 1 year ago

There's a header capitalisation issue between Py2 (works) and Py 3 (fails):


--- old/youtube_dl/extractor/itv.py
+++ new/youtube_dl/extractor/itv.py
@@ -59,7 +59,7 @@ class ITVBaseIE(InfoExtractor):

     @staticmethod
     def _vanilla_ua_header():
-        return {'User-agent': 'Mozilla/5.0'}
+        return {'User-Agent': 'Mozilla/5.0'}

     def _download_webpage_handle(self, url, video_id, *args, **kwargs):
         # specialised to (a) use vanilla UA (b) detect geo-block
xtechgamer735 commented 1 year ago

Well spotted! Manually defining that header (--add-header User-Agent:Mozilla/5.0) is working perfectly for me now. Thanks for your help.

Vangelis66 commented 1 year ago

Using the latest daily/nightly Windows build, youtube-dl.exe, which is built on py3.4, the download works (for me) if the default external downloader (ffmpeg) is used for fragment download and decryption:

yt-dl -v --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[debug] ffmpeg command line: ".\FFmpeg\ffmpeg" -y -loglevel verbose -v 8 -stats -headers 
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
User-agent: Mozilla/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3663.2 Safari/537.36
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Cookie: (removed for brevity)"
 -i "https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8" -c copy -fs 10241 -f mp4 "-bsf:a" aac_adtstoasc "file:Heathrow - Britain'sBusiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4.part"
ffmpeg version n6.1-dev-705-N-110589-g30c71ef Copyright (c) 2000-2023 the FFmpeg developers
  built with Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31944 for x86

  configuration: (removed for brevity)
  libavutil      58.  7.100 / 58.  7.100
  libavcodec     60. 11.100 / 60. 11.100
  libavformat    60.  5.100 / 60.  5.100
  libavdevice    60.  2.100 / 60.  2.100
  libavfilter     9.  8.100 /  9.  8.100
  libswscale      7.  2.100 /  7.  2.100
  libswresample   4. 11.100 /  4. 11.100
frame=    0 fps=0.0 q=-1.0 size=       0kB time=00:00:00.03 bitrate=   0.0kbits/
frame=   32 fps=0.0 q=-1.0 Lsize=     262kB time=00:00:01.51 bitrate=1420.5kbits/s speed= 9.7x
[ffmpeg] Downloaded 268706 bytes
[download] 100% of 262.41KiB in 00:03

The reason is found below:

User-agent: Mozilla/5.0
...
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3663.2 Safari/537.36

i.e. while itv.py itself injects "User-agent", ffmpeg in turn injects "User-Agent" πŸ˜‰ ...

OTOH, delegating HLSe download and decryption to [hlsnative] (implies built-in crypto support) will fail 😞 , as reported above:

yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--hls-prefer-native', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 1992, in process_info
  File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 1931, in dl
  File "D:\a\youtube-dl\youtube-dl\youtube_dl\downloader\common.py", line 376, in download
  File "D:\a\youtube-dl\youtube-dl\youtube_dl\downloader\hls.py", line 64, in real_download
  File "D:\a\youtube-dl\youtube-dl\youtube_dl\YoutubeDL.py", line 2355, in urlopen
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 470, in open
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 580, in http_response
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 508, in error
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 442, in _call_chain
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\urllib\request.py", line 588, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

It has to be noted this is a recent-ish ITV change 😠 that affects specifically the variant playlists (but not the master one), because I did use the "fixed" itvIE back in mid-February of this year (as bundled inside the py3.4 youtube-dl-exe) to test-fetch some programmes with [hlsnative], and at the time "a" vs "A" wasn't an issue...

@dirkf , any plans to merge the patch to master ?

... With the patch applied and [hlsnative]:

yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065"

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-v', '--hls-prefer-native', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.22.334
[debug] Python 3.4.4 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2d 9 Jul 2015, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 465
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[download] 100% of 10.00KiB in 00:01
[debug] ffmpeg command line: ".\FFmpeg\ffprobe" -show_streams "file:Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4"

... Many thanks for your unwavering efforts! πŸ‘

dirkf commented 1 year ago

With my current dev version:

$ python3.5 -m pip list 2>&1 | fgrep crypto
cryptography                 3.3.2
pycrypto                     2.6.1
pycryptodomex                3.11.0
$ python3.5 -m youtube_dl -v --test 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065' --hls-prefer-native
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--test', 'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065', '--hls-prefer-native']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: f55ab2716
[debug] Python 3.5.2 (CPython 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial (OpenSSL 1.0.2n  7 Dec 2017, glibc 2.9)
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 465
[download] Destination: Heathrow - Britain's Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4
[download] 100% of 10.00KiB in 00:00
[debug] ffmpeg command line: ffprobe -show_streams 'file:Heathrow - Britain'"'"'s Busiest Airport - Series 8 - Episode 2 - Episode 2-2a3168a0065.mp4'
$ 

So clearly an update to master is needed.

Vangelis66 commented 1 year ago

There's a header capitalisation issue between Py2 (works) and Py 3 (fails):

In addition to my previous post above, I find above statement not to be entirely true, at least under Windows πŸ˜‰ (?) ...

Using my own compiled youtube-dl.exe, built with CPython 2.7.18+PyCrypto-2.6.1, I find that the current itvIE with "User-agent" doesn't work even in the case of Py2, when [hlsnative] is specified:

yt-dl -v --hls-prefer-native --test "https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--ffmpeg-location', u'.\\FFmpeg', u'--external-downloader-args', u'-v 8 -stats', u'-v', u'--hls-prefer-native', u'--test', u'https://www.itv.com/watch/heathrow-britains-busiest-airport/2a3168/2a3168a0065']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.05.11
[debug] Lazy loading extractors enabled
[debug] Python 2.7.18 (CPython 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.0.2t  10 Sep 2019, -)
[debug] exe versions: ffmpeg n6.1-dev-705-N-110589-g30c71ef, ffprobe n6.1-dev-705-N-110589-g30c71ef, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[ITV] 2a3168a0065: Downloading webpage
[ITV] 2a3168a0065: Downloading JSON metadata
[ITV] 2a3168a0065: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://itvpnpmobile.blue.content.itv.com/2-3168-0065-001/6/1/VAR003/2-3168-0065-001_6_1_VAR003.ism/2-3168-0065-001_6_1_VAR003-audio=96000-video=1703877.m3u8'
[hlsnative] Downloading m3u8 manifest
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "youtube_dl\YoutubeDL.pyo", line 1992, in process_info
  File "youtube_dl\YoutubeDL.pyo", line 1931, in dl
  File "youtube_dl\downloader\common.pyo", line 376, in download
  File "youtube_dl\downloader\hls.pyo", line 64, in real_download
  File "youtube_dl\YoutubeDL.pyo", line 2355, in urlopen
  File "urllib2.pyo", line 435, in open
  File "urllib2.pyo", line 548, in http_response
  File "urllib2.pyo", line 473, in error
  File "urllib2.pyo", line 407, in _call_chain
  File "urllib2.pyo", line 556, in http_error_default
HTTPError: HTTP Error 403: Forbidden

dirkf's log above demonstrates download success under py2.7 with ffmpeg as the HLSe downloader...

OP (under MacOS, though) does use ffmpeg-6.0 as the default (external) downloader (py3.11); in his debug log however, I see that the ffmpeg provided "User-Agent" is injected/sent first, whereas the itv.py provided "User-agent" is injected/sent second; in the logs provided by dirkf (ffmpeg-4.3, py2.7) and myself (ffmpeg-n6.1-dev, py3.4), the opposite is true, so perhaps that's the deciding factor in this bug?

IMHO, this bug should be re-opened, until properly mitigated by a relevant PR or master branch commit...

dirkf commented 1 year ago

The dev version works equally with Py 2.7 where

$ python2.7 -m pip list 2>&1 | fgrep crypto
pycrypto                           2.6.1
pycryptodomex                      3.4.3
$

The patch at https://github.com/ytdl-org/youtube-dl/issues/32217#issuecomment-1556264918 is not the only place where capitalisation went wrong.

Vangelis66 commented 1 year ago

https://github.com/ytdl-org/youtube-dl/blob/d1c6c5c4d618fa950813c0c71aede34a5ac851e9/youtube_dl/extractor/itv.py#L72

?