Closed HellerCat closed 6 years ago
Can confirm problem with youtube-dl 2018.03.20. I'm also getting the warning: "Unable to download JSON metadata: HTTP Error 403: Forbidden". I tried a remix with rtmpdump and got this response:
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--restrict-filenames', u'--recode-video', u'mp4', u'--postprocessor-args', u'-codec copy', u'--verbose', u'-o', u'%(series)s-%(season)s-%(title)s.%(ext)s', u'https://www.itv.com/hub/vera/1a7314a0034']
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2018.03.20
[debug] Python version 2.7.10 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.2-tessus, ffprobe 3.4.2-tessus, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 25.234.179.76 (GB) as X-Forwarded-For.
[ITV] 1a7314a0034: Downloading webpage
[ITV] 1a7314a0034: Downloading XML
[ITV] 1a7314a0034: Downloading JSON metadata
WARNING: Unable to download JSON metadata: HTTP Error 403: Forbidden
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'rtmpe://cp72511.edgefcs.net/ondemand'
[download] Destination: Vera-Season_8-Darkwater.flv
[debug] rtmpdump command line: rtmpdump --verbose -r 'rtmpe://cp72511.edgefcs.net/ondemand' -o Vera-Season_8-Darkwater.flv.part --swfVfy 'http://www.itv.com/mercury/Mercury_VideoPlayer.swf' --pageUrl 'https://www.itv.com/hub/vera/1a7314a0034' --app 'ondemand?auth=eaEaQaGcIcFcFbsc4dzdfaTbBaMdqdJbHdJblbOdqd0bIana4bkb0bTdLdOcUaYcUaR-bASTQG-eS-6xo3GColwsEr&aifp=v002&slist=production/' --playpath 'mp4:production/priority/CATCHUP/1/7314/0034/001/1-7314-0034-001_PC011800_16X9.mp4' --resume --skip 1
[rtmpdump] RTMPDump 2.4
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 2
[rtmpdump] DEBUG: Parsed host : cp72511.edgefcs.net
[rtmpdump] DEBUG: Parsed app : ondemand
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMPE
[rtmpdump] DEBUG: Hostname : cp72511.edgefcs.net
[rtmpdump] DEBUG: Port : 1935
[rtmpdump] DEBUG: Playpath : mp4:production/priority/CATCHUP/1/7314/0034/001/1-7314-0034-001_PC011800_16X9.mp4
[rtmpdump] DEBUG: tcUrl : rtmpe://cp72511.edgefcs.net:1935/ondemand?auth=eaEaQaGcIcFcFbsc4dzdfaTbBaMdqdJbHdJblbOdqd0bIana4bkb0bTdLdOcUaYcUaR-bASTQG-eS-6xo3GColwsEr&aifp=v002&slist=production/
[rtmpdump] DEBUG: swfUrl : http://www.itv.com/mercury/Mercury_VideoPlayer.swf
[rtmpdump] DEBUG: pageUrl : https://www.itv.com/hub/vera/1a7314a0034
[rtmpdump] DEBUG: app : ondemand?auth=eaEaQaGcIcFcFbsc4dzdfaTbBaMdqdJbHdJblbOdqd0bIana4bkb0bTdLdOcUaYcUaR-bASTQG-eS-6xo3GColwsEr&aifp=v002&slist=production/
[rtmpdump] DEBUG: live : no
[rtmpdump] DEBUG: timeout : 30 sec
[rtmpdump] DEBUG: SWFSHA256:
[rtmpdump] DEBUG: b6 c8 96 6d a3 f4 96 10 be 71 78 b0 1c a3 3d 04
[rtmpdump] DEBUG: 6b bf 91 5e 29 08 d9 da fe 11 e4 b0 42 d8 ee ea
[rtmpdump] DEBUG: SWFSize : 990750
[rtmpdump] DEBUG: Setting buffer time to: 36000000ms
[rtmpdump] Connecting ...
[rtmpdump] DEBUG: RTMP_Connect1, ... connected, handshaking
[rtmpdump] DEBUG: HandShake: Client type: 06
[rtmpdump] DEBUG: HandShake: DH pubkey position: 472
[rtmpdump] DEBUG: HandShake: Client digest offset: 1204
[rtmpdump] DEBUG: HandShake: Initial client digest:
[rtmpdump] DEBUG: df 24 68 cf 85 81 b7 74 49 db a3 ac 4b c0 21 89
[rtmpdump] DEBUG: d6 5a 2c b5 37 2d 93 6d 1f 0a ae 3a ef 71 34 ff
[rtmpdump] DEBUG: HandShake: Type Answer : 08
[rtmpdump] WARNING: HandShake: Type mismatch: client sent 6, server answered 8
[rtmpdump] DEBUG: HandShake: Server Uptime : 711087739
[rtmpdump] DEBUG: HandShake: FMS Version : 5.0.7.1
[rtmpdump] DEBUG: HandShake: Server DH public key offset: 342
[rtmpdump] DEBUG: HandShake: Secret key:
[rtmpdump] DEBUG: 01 83 8b e0 5d 5c b6 9f 28 70 cf 6d 7e 58 1b a4
[rtmpdump] DEBUG: 7e 0d 2d 1d 34 b8 db 43 d4 34 ee 63 cf ff 0e 5f
[rtmpdump] DEBUG: 74 3b 2e 21 ae bf f0 1b 09 f6 10 30 3e 0f 2e d8
[rtmpdump] DEBUG: 2d ec de af 1d 77 c0 03 b1 71 5c a4 d7 44 86 55
[rtmpdump] DEBUG: f8 ff 65 f7 24 7b 69 3d 09 20 38 70 7c 70 9a 16
[rtmpdump] DEBUG: d2 51 85 b2 85 44 02 71 63 3a e9 02 f9 ab 09 04
[rtmpdump] DEBUG: ec f2 4a 5f de 9c e9 cf c9 da e1 87 fc 53 13 59
[rtmpdump] DEBUG: 63 b1 63 25 2c 5d 09 d2 a5 7d 93 e4 4f 82 fd 98
[rtmpdump] DEBUG: RC4 Out Key:
[rtmpdump] DEBUG: a4 6c bb 45 0e 60 4a ea b6 7a 59 e4 fc ef f3 ef
[rtmpdump] DEBUG: RC4 In Key:
[rtmpdump] DEBUG: 87 93 d8 61 8a 25 3a 8e 44 73 4c 26 86 be 9b da
[rtmpdump] DEBUG: HandShake: Calculated digest key from secure key and server digest:
[rtmpdump] DEBUG: 5f 88 95 89 59 00 eb 65 fa e7 a4 c3 23 f5 11 be
[rtmpdump] DEBUG: 57 5e 6d 81 f1 6b 0e 5f 0f 7a 23 ce 82 06 e6 e1
[rtmpdump] DEBUG: HandShake: Client signature calculated:
[rtmpdump] DEBUG: 5e 85 43 0f 18 18 da 2e 6c c1 0a 12 32 ff 03 9f
[rtmpdump] DEBUG: 99 c4 74 97 e2 ac 02 71 1a c6 93 a5 b7 82 9c b7
[rtmpdump] DEBUG: HandShake: Server sent signature:
[rtmpdump] DEBUG: c8 df b8 4e 28 2b 6a 1c 9a c1 2a 98 b7 58 86 fd
[rtmpdump] DEBUG: 5d 0e b9 86 cb e0 b1 5f 3a a8 a9 f9 61 fe fa bd
[rtmpdump] DEBUG: HandShake: Digest key:
[rtmpdump] DEBUG: 6c 45 7b d0 8e e7 2d 40 0d 41 a7 a6 ca 15 2b e1
[rtmpdump] DEBUG: d8 80 39 9d fd 6e 3c 39 03 d9 93 25 00 9b bf dd
[rtmpdump] DEBUG: HandShake: Signature calculated:
[rtmpdump] DEBUG: c8 df b8 4e 28 2b 6a 1c 9a c1 2a 98 b7 58 86 fd
[rtmpdump] DEBUG: 5d 0e b9 86 cb e0 b1 5f 3a a8 a9 f9 61 fe fa bd
[rtmpdump] DEBUG: HandShake: Genuine Adobe Flash Media Server
[rtmpdump] DEBUG: HandShake: Handshaking finished....
[rtmpdump] DEBUG: RTMP_Connect1, handshaked
[rtmpdump] DEBUG: Invoking connect
[rtmpdump] INFO: Connected...
[rtmpdump] DEBUG: RTMPSockBuf_Fill, recv returned -1. GetSockError(): 35 (Resource temporarily unavailable)
[rtmpdump] ERROR: RTMP_ReadPacket, failed to read RTMP packet header
[rtmpdump] DEBUG: Closing connection.
[rtmpdump] 0 bytes
[rtmpdump] RTMPDump 2.4
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 2
[rtmpdump] DEBUG: Parsed host : cp72511.edgefcs.net
[rtmpdump] DEBUG: Parsed app : ondemand
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMPE
[rtmpdump] DEBUG: Hostname : cp72511.edgefcs.net
[rtmpdump] DEBUG: Port : 1935
[rtmpdump] DEBUG: Playpath : mp4:production/priority/CATCHUP/1/7314/0034/001/1-7314-0034-001_PC011800_16X9.mp4
[rtmpdump] DEBUG: tcUrl : rtmpe://cp72511.edgefcs.net:1935/ondemand?auth=eaEaQaGcIcFcFbsc4dzdfaTbBaMdqdJbHdJblbOdqd0bIana4bkb0bTdLdOcUaYcUaR-bASTQG-eS-6xo3GColwsEr&aifp=v002&slist=production/
[rtmpdump] DEBUG: swfUrl : http://www.itv.com/mercury/Mercury_VideoPlayer.swf
[rtmpdump] DEBUG: pageUrl : https://www.itv.com/hub/vera/1a7314a0034
[rtmpdump] DEBUG: app : ondemand?auth=eaEaQaGcIcFcFbsc4dzdfaTbBaMdqdJbHdJblbOdqd0bIana4bkb0bTdLdOcUaYcUaR-bASTQG-eS-6xo3GColwsEr&aifp=v002&slist=production/
[rtmpdump] DEBUG: live : no
[rtmpdump] DEBUG: timeout : 30 sec
[rtmpdump] DEBUG: SWFSHA256:
[rtmpdump] DEBUG: b6 c8 96 6d a3 f4 96 10 be 71 78 b0 1c a3 3d 04
[rtmpdump] DEBUG: 6b bf 91 5e 29 08 d9 da fe 11 e4 b0 42 d8 ee ea
[rtmpdump] DEBUG: SWFSize : 990750
[rtmpdump] DEBUG: Failed to get last keyframe.
[rtmpdump] DEBUG: Closing connection.
ERROR: rtmpdump exited with code 1
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
youtube_dl.main()
File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 471, in main
_real_main(argv)
File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 461, in _real_main
retcode = ydl.download(all_urls)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1989, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 850, in process_ie_result
return self.process_video_result(ie_result, download=download)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1623, in process_video_result
self.process_info(new_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1896, in process_info
success = dl(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1835, in dl
return fd.download(name, info)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
return self.real_download(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/rtmp.py", line 202, in real_download
self.report_error('rtmpdump exited with code %d' % retval)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 166, in report_error
self.ydl.report_error(*args, **kargs)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 613, in report_error
self.trouble(error_message, tb)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 575, in trouble
tb_data = traceback.format_list(traceback.extract_stack())
Could this be a geo-blocking change ?
@section83 wrote:
Could this be a geo-blocking change ?
It's exactly THAT! The ITV Hub people are now playing hard ball...
I'm also getting the warning: "Unable to download JSON metadata: HTTP Error 403: Forbidden"
"Techie" details:
With the (legacy) RTMPE (Flash) streams, there's only a soft geo-block on the Playlist (stream data) acquisition URL:
http://mercury.itv.com/PlaylistService.svc?wsdl
or https://secure-mercury.itv.com/PlaylistService.svc?wsdl
That soft block can be lifted via an X-Forwarded-For header trick, the FMS server itself doesn't geo-fence (yet?); for your show in question, on Windows (Vista SP2 x86) and from an overseas location, latest yt-dl (2018.03.20) shows:
youtube-dl -F "https://www.itv.com/hub/vera/1a7314a0034"
[ITV] 1a7314a0034: Downloading webpage
[ITV] 1a7314a0034: Downloading XML
[ITV] 1a7314a0034: Downloading JSON metadata
[ITV] 1a7314a0034: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[info] Available formats for 1a7314a0034:
format code extension resolution note
rtmp-600 flv unknown 600k
rtmp-800 flv unknown 800k
rtmp-1200 flv unknown 1200k
rtmp-1500 flv unknown 1500k
rtmp-1800 flv unknown 1800k (best)
i.e. only the RTMPE streams are detected; on my setup/location, I have no issues fetching the best RTMPE stream (-f=rtmp-1800 - uses a DIRECT connection):
[ITV] 1a7314a0034: Downloading webpage
[ITV] 1a7314a0034: Downloading XML
[ITV] 1a7314a0034: Downloading JSON metadata
[ITV] 1a7314a0034: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[download] Destination: Darkwater-1a7314a0034.flv
[download] 2.7% of ~1.03GiB at 1.06MiB/s ETA 16:04
You appear to be on a Mac; what exactly do you mean by
I tried a remix with rtmpdump
Me thinks your issue lies in a deficient RTMPDump binary and/or your connection settings; remember, for RTMP streams you need default port 1935 open! Find/compile a new RTMPDump MacOS binary and retry; sorry I can't help, no Macs here...
Apart from the legacy RTMPE streams, ITV also provide new encrypted MPEG-DASH streams (for desktop browsers) and encrypted AppleHLS streams (for iDevices); the former are of no use to yt-dl, because common encryption (cenc) hasn't been broken; the latter use AES-128 encryption, that yt-dl/FFmpeg can cope with...
The new streams employ a new playlist API service:
https://magni.itv.com/playlist/itvonline/*
that is still being geo-fenced but, yet again, this soft block can be lifted with an X-F-F trick... Up until some days ago, that URI yielded HLS master playlists (.m3u8) hosted on a "itvpnpmobile-a.akamaihd.net" host that didn't geo-block; as a result, yt-dl could still detect/fetch those HLS streams from overseas... Around Mar 20th 2018, that host has been reconfigured to block non-UK IPs (geo-fencing), but this one is a "hard" block, meaning an X-F-F trick won't work ! This is the reason you get the WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
, ergo no HLS streams from overseas...
TL;DR
Overseas users: Only RTMPE streams will be detected from now on, for as long as they remain available (they are deprecated by ITV and not always available for their entire VOD catalogue; i.e., some shows are missing RTMPE streams). You should still be able to fetch those streams from overseas, provided you have a functional rtmpdump binary...
In the case of ITV shows lacking downloadable RTMPE streams (like the OP's
https://www.itv.com/hub/action-team/2a5066a0004
), then, if overseas, you're in a predicament; my first advice would be to wait for some days after the date of broadcast, in the hope of ITV providing the missing Flash streams (that you can fetch from overseas); however, if you're in a hurry or the RTMPE streams never show up, you'd have to resort to more drastic geo-location circumvention methods (UK HTTP proxy, full blown UK VPN, SmartDNS service with support for ITV HLS, etc), because not only the master playlist and its variants are geo-fenced, so are individual HLS fragments; IOW, you'd have to route all HTTP traffic from itvpnpmobile-a.akamaihd.net to the proxy/VPN... And it won't be long before ITV start blacklisting commercial proxy/VPN IPs (we saw that already with BBC iPlayer)...
e.g. using a UK VPN, @section83's show now yields:
youtube-dl -F "https://www.itv.com/hub/vera/1a7314a0034"
[ITV] 1a7314a0034: Downloading webpage
[ITV] 1a7314a0034: Downloading XML
[ITV] 1a7314a0034: Downloading JSON metadata
[ITV] 1a7314a0034: Downloading m3u8 information
[info] Available formats for 1a7314a0034:
format code extension resolution note
hls-136 mp4 audio only 136k , mp4a.40.2@128k
hls-583 mp4 512x288 583k , avc1.64001F@ 421k, 25.0fps, mp4a.40.2@128k
rtmp-600 flv unknown 600k
hls-777 mp4 512x288 777k , avc1.64001F@ 604k, 25.0fps, mp4a.40.2@128k
rtmp-800 flv unknown 800k
hls-1029 mp4 896x504 1029k , avc1.64001F@ 841k, 25.0fps, mp4a.40.2@128k
rtmp-1200 flv unknown 1200k
hls-1337 mp4 896x504 1337k , avc1.64001F@1132k, 25.0fps, mp4a.40.2@128k
rtmp-1500 flv unknown 1500k
hls-1629 mp4 896x504 1629k , avc1.64001F@1408k, 25.0fps, mp4a.40.2@128k
rtmp-1800 flv unknown 1800k (best)
(notice how the HLS streams now appear) and @HellerCat's one now includes downloadable HLS streams:
youtube-dl -F "https://www.itv.com/hub/action-team/2a5066a0004"
[ITV] 2a5066a0004: Downloading webpage
[ITV] 2a5066a0004: Downloading XML
[ITV] 2a5066a0004: Downloading JSON metadata
[ITV] 2a5066a0004: Downloading m3u8 information
[info] Available formats for 2a5066a0004:
format code extension resolution note
hls-136 mp4 audio only 136k , mp4a.40.2@128k
hls-585 mp4 512x288 585k , avc1.64001F@ 423k, 25.0fps, mp4a.40.2@128k
hls-774 mp4 512x288 774k , avc1.64001F@ 601k, 25.0fps, mp4a.40.2@128k
hls-1021 mp4 896x504 1021k , avc1.64001F@ 834k, 25.0fps, mp4a.40.2@128k
hls-1313 mp4 896x504 1313k , avc1.64001F@1109k, 25.0fps, mp4a.40.2@128k
hls-1590 mp4 896x504 1590k , avc1.64001F@1371k, 25.0fps, mp4a.40.2@128k (best)
Many thanks for all that. I'm happy not to work around geo-blocking so, I'll leave things as they are.
I tried a remix with rtmpdump
That was just a brain fade - I meant "remuxing" but Apple Safari auto-corrected it and I wasn't remuxing anyway.
Please follow the guide below
x
into all the boxes [ ] relevant to your issue (like this:[x]
)Make sure you are using the latest version: run
youtube-dl --version
and ensure your version is 2018.03.20. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.Before submitting an issue make sure you have:
What is the purpose of your issue?
The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue
If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the
-v
flag to your command line you run youtube-dl with (youtube-dl -v <your command line>
), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):
Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
Description of your issue, suggested solution and other information
Hi all, I am no longer able to record items from ITV Hub. For example, https://www.itv.com/hub/action-team/2a5066a0004. I was able to record them last week, but the log file says "No video formats found". I upgraded to youtube-dl 2018.03.20 in the last few days. Unfortunately, I cannot remember if I have been able to record anything on ITV Hub with this version.
I hope I have filled in issue form correctly and that my issue is clear. If not please let me know.
Many Thanks M