Support for channel4.com OnDemand

canavan commented 10 years ago

Please add support for channel 4 OnDemand, e.g. http://www.channel4.com/programmes/the-it-crowd/4od

There's already a separate tool (in ruby, uses rtmpdump) that supports downloading from that service: https://github.com/robwatkins/4od-dl

youtube-dl --verbose http://www.channel4.com/programmes/the-it-crowd/4od [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['--verbose', 'http://www.channel4.com/programmes/the-it-crowd/4od'] [debug] Encodings: locale 'ISO8859-1', fs 'ISO8859-1', out 'ISO8859-1', pref: 'ISO8859-1' [debug] youtube-dl version 2013.12.26 [debug] Python version 2.7.3 - IRIX64-6.5-IP35-mips-n32bit-ELF [debug] Proxy map: {} WARNING: Falling back on generic information extractor. [generic] 4od: Downloading webpage [generic] 4od: Extracting information ERROR: Unsupported URL: http://www.channel4.com/programmes/the-it-crowd/4od; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type youtube-dl -U to update. Traceback (most recent call last): File "/usr/nekoware/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 484, in extract_info ie_result = ie.extract(url) File "/usr/nekoware/bin/youtube-dl/youtube_dl/extractor/common.py", line 149, in extract return self._real_extract(url) File "/usr/nekoware/bin/youtube-dl/youtube_dl/extractor/generic.py", line 330, in _real_extract raise ExtractorError(u'Unsupported URL: %s' % url) ExtractorError: Unsupported URL: http://www.channel4.com/programmes/the-it-crowd/4od; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type youtube-dl -U to update.

fstirlitz commented 9 years ago

Downloading itself seems easy; the stream is a regular f4m, and finding the stream's URL is not difficult. The only problem is that the stream is encrypted by Flash Access DRM.

Some resources:

F4V spec: http://download.macromedia.com/f4v/video_file_format_spec_v10_1.pdf
https://code.google.com/p/get-flash-videos/issues/detail?id=379
https://www.kernel.org/pub/scm/linux/kernel/git/jejb/get-flash-videos.git/ seems to have some working code (lib/FlashVideo/F4VDownloader.pm); the only missing piece is obtaining the decryption key.

jleclanche commented 8 years ago

@fstirlitz isn't that what AdobeHDS.php implements?

dstftw commented 8 years ago

@jleclanche it does not support DRM.

jleclanche commented 8 years ago

Ah fair enough

ppsspp213 commented 8 years ago

Any insight with thiis???

http://www.channel4.com/programmes/formula-1/on-demand/63760-055

[code] http://ll.abrstream.channel4.com/CH4_08_02_24_63760055001001_001_1500000_.f4m

streamUri>http://ll.abrstream.channel4.com/CH4_08_02_24_63760055001001_001_1500000_.f4m

token>jZGfXUrmGGW2uXKUQFIjgBlHc4hA0t9QZqvEDPwjNrnzUrUVVvKcgXEWG6eSiP1cEKIpqTL4J6o1 +dMCZOQpZkUWedsnyeGjSFYY4SarrsS1yICL2gkjbzdP/C4kBf22cmmN7cux+RwmjWuA5jKEtpUXGn3Ec3Ey6MM5qQMQH7wc2VnB/Erp4oC94gTHNLb

e>1473046553 transcodeId>1587713 brand4oDAtomUrl>https://pmlsc.channel4.com/pmlsd/formula-1/4od.atom

programmeId>63760/055

GET /CH4_08_02_24_63760055001001_0011500000.f4m HTTP/1.1 Host: ll.abrstream.channel4.com User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 7 Build/MMB29V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36

OPR/36.0.2126.101812 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Language: en-GB,en;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Referer: http://www.channel4.com/all4fplayer/1.0.160/4odplayer_bips_r.swf Cookie: c4_cookie_policy=A;

C4_Identity="eyJkaXNwbGF5TmFtZSI6IkNoYXJsZXMiLCJnZW5kZXIiOiJGRU1BTEUiLCJhZ2UiOiI2NCIsInVzZXJJZCI6IjdmN2M4MzEzLWMwMDctNDQ3Mi05NGY2LWExN2I2YzI0O

TUxYyIsInR5cGUiOiJDNCIsInJlbWVtYmVyTWUiOiJ0cnVlIiwiY3JlYXRlZCI6IjE0NzMwMzk1NzE4ODYiLCJoYXNoIjoiTUhnZUFMWWRfWjR1aU1qdFBkdE1aUSIsImFkZHJlc3NDYXB

0dXJlZCI6InRydWUifQ=="; s_fid=50051E692397AF16-176ED4E71FF0A632; s_vnum=1475631616357%26vn%3D1; s_vi=[CS]v1|2BE666720531251D-60000109A000A811

[CE];

4nav_Identity=eyJkaXNwbGF5TmFtZSI6IkNoYXJsZXMiLCJnZW5kZXIiOiJGRU1BTEUiLCJhZ2UiOiI2NCIsInVzZXJJZCI6IjdmN2M4MzEzLWMwMDctNDQ3Mi05NGY2LWExN2I2YzI0

OTUxYyIsInR5cGUiOiJDNCIsInJlbWVtYmVyTWUiOiJ0cnVlIiwiY3JlYXRlZCI6IjE0NzMwMzk1NzE4ODYiLCJoYXNoIjoiTUhnZUFMWWRfWjR1aU1qdFBkdE1aUSIsImFkZHJlc3NDYX

B0dXJlZCI6InRydWUifQ%3D%3D; ADRUM=s=1473042976492&r=http%3A%2F%2Fwww.channel4.com%2Fprogrammes%2Fformula-1%2Fon-demand%2F63760-055%3F0;

omn_intcmp=categories_sport_popular Connection: keep-alive [/code]

suntong commented 7 years ago

@ppsspp213 May I know how you get above pls? Thx.

Oshawk commented 6 years ago

Hello,

I may have some, all be it a small amount of progress on this via the android download method. I will attach a .saz file with a few captures that I managed to get, one appears to be programme info, another subtitles and there are also some which seem to be audio only versions of the programme although unfortunately, I can't seem to get them to play.

Can any of you progress with this? (Note, I removed most of the data from the capture file but I think I included most useful stuff.)

channel4.zip

ppsspp213 commented 6 years ago

You can capture the audio with Replay Media Catcher though i'm not sure if it just records what is playing or if it dls the actual packets.

There was a fault in earlier versions of chrome that has since been patched that allowed you capture the video unencrypted. On 7 Jan 2018 4:07 p.m., "Oshawk" notifications@github.com wrote:

Hello,

I may have some, all be it a small amount of progress on this via the android download method. I will attach a .saz file with a few captures that I managed to get, one appears to be programme info, another subtitles and there are also some which seem to be audio only versions of the programme although unfortunately, I can't seem to get them to play.

Can any of you progress with this? (Note, I removed most of the data from the capture file but I think I included most useful stuff.)

channel4.zip https://github.com/rg3/youtube-dl/files/1609925/channel4.zip

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rg3/youtube-dl/issues/2050#issuecomment-355832948, or mute the thread https://github.com/notifications/unsubscribe-auth/AF53ZEr5L8Y6WWg2qSTA6kBNMG5PVZ2aks5tIOutgaJpZM4BWWvQ .

Oshawk commented 6 years ago

@ppsspp213 Okay, I still have confidence that the download method may get us somewhere, I will try and capture a full one and see what happens.

lozbrown commented 6 years ago

+1 is there any hope for f4m drm?

darthhaggis commented 6 years ago

Has there been any updates or support added for this request?

codingHahn commented 5 years ago

Updates @Oshawk ?

ghost commented 5 years ago

It no longer uses flash, now using html5 Is it possible to make it support?

superclarkk commented 5 years ago

An answer to @BMD4E 's question would be awesome

lozbrown commented 5 years ago

Really ?

superclarkk commented 5 years ago

The source is this article. I just phoned Channel4, and they confirmed they're migrating to HTML5 but, because they have such a large back catalogue, it's taking a while and they don't have a completion date.

ppsspp213 commented 5 years ago

Have they said if it’s going to 720p, 1080p or 480p for content from 2006/8 onwards?

On Sat, 27 Jul 2019 at 12:43 pm, superclarkk notifications@github.com wrote:

The source is this article http://www.csimagazine.com/csi/Channel4-migrates-to-HTML5-with-Adobe-Primetime.php. I just phoned Channel4, and they confirmed they're migrating to HTML5 but, because they have such a large back catalogue, it's taking a while and they don't have a completion date.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/2050?email_source=notifications&email_token=ABPHOZD6TYC3RE7IWX2NEG3QBQYFJA5CNFSM4AKZNPIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26J5LA#issuecomment-515677868, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPHOZGHZP6DPOCRI3XI5F3QBQYFJANCNFSM4AKZNPIA .

superclarkk commented 5 years ago

They were pretty cagey, so the above was all the info I could get.

ppsspp213 commented 5 years ago

Ok, thanks for the update 👍

On Sat, 27 Jul 2019 at 1:29 pm, superclarkk notifications@github.com wrote:

They were pretty cagey, so the above was all the info I could get.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/2050?email_source=notifications&email_token=ABPHOZGO5VOOQT3NXHH3SG3QBQ5RTA5CNFSM4AKZNPIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26KRQA#issuecomment-515680448, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPHOZESU6UN3TFCWH7S45DQBQ5RTANCNFSM4AKZNPIA .

Vangelis66 commented 5 years ago

They're still going to use

strong DRM based content protection across the feature set

in their migration to the HTML5-based All4 web player, so I don't quite understand the latent excitement here over that change...

The selection of Primetime allows us to look at alignment with the other Adobe products that we currently use

https://theblog.adobe.com/transition-to-html5/

The title of this long-standing issue is "Support for channel4.com OnDemand", surest bet is that won't arrive with the web player format overhaul... 😞 Just my 2p, ofc...

ppsspp213 commented 5 years ago

I’m not a fan of their adaptive video quality streaming, where if it drops packets it reduces the quality downwards, it’d be much better if they produced a codec with more EDC imo.

On Sat, 27 Jul 2019 at 11:48 pm, Vangelis66 notifications@github.com wrote:

They're still going to use

strong DRM based content protection across the feature set

in their migration to the HTML5-based All4 web player, so I don't quite understand the latent excitement here over that change...

The selection of Primetime allows us to look at alignment with the other Adobe products that we currently use

https://theblog.adobe.com/transition-to-html5/

The title of this long-standing issue is "Support for channel4.com http://channel4.com OnDemand", surest bet is that won't arrive with the web player format overhaul... 😞 Just my 2p, ofc...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ytdl-org/youtube-dl/issues/2050?email_source=notifications&email_token=ABPHOZD2WGHXKECW6RBVLHLQBTGCZA5CNFSM4AKZNPIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26T4KQ#issuecomment-515718698, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPHOZDBYN7UL66C2PGMFHLQBTGCZANCNFSM4AKZNPIA .

theshillito commented 3 years ago

Getting a slightly different output these days.

youtube-dl.exe --verbose https://www.channel4.com/programmes/big-fat-quiz/on-demand/71755-001
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.channel4.com/programmes/big-fat-quiz/on-demand/71755-001']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2020.12.29
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[generic] 71755-001: Requesting header
WARNING: Could not send HEAD request to https://www.channel4.com/programmes/big-fat-quiz/on-demand/71755-001: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>
[generic] 71755-001: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)'),))
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk6o\build\youtube_dl\extractor\common.py", line 632, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk6o\build\youtube_dl\YoutubeDL.py", line 2248, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 464, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 482, in _open
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk6o\build\youtube_dl\utils.py", line 2736, in https_open
  File "C:\Python\Python34\lib\urllib\request.py", line 1185, in do_open

Vangelis66 commented 3 years ago

@theshillito

Nothing new has cropped up with regards to All4, i.e. they're still using DRM encrypted streams, which yt-dl can't process... 😭

What you have reported above is a local issue you're seeing when the official yt-dl executable tries to connect securely (over HTTPS) with www.channel4.com ...

The root cause of that issue is the same as with the numerous duplicate issues where Windows users can't update the youtube-dl.exe file past May 30th 2020...

TL:DR: youtube-dl.exe is built on Python 3.4.4, which bundles modules containing an older version of the openssl crypto library; that older version has a bug (fixed in later versions, hence not present in more recent Python).

This is somewhat technical, but when an expired intermediate certificate is present inside your Windows cert store, older openssl (in Py3.4.4) will select this expired one and abort secure connection to www.channel4.com, instead of trying to complete the certificate chain to the server via a different - but valid - route...

Workarounds: Use the somewhat unsafe flag --no-check-certificate, or deal with the issue directly by

Loading https://www.channel4.com/ in a Microsoft browser, like Internet Explorer 11 or Edge; the browser will fetch all needed (valid) intermediate certs and will cache them in the Windows CA store (this is where youtube-dl.exe looks by default for certs).
In your Windows CA store, locate and remove all expired INTERMEDIATE certificates; you have better leave the expired ROOT certificates (if any) alone, though...

Then, youtube-dl.exe (on Py3.4) will have no trouble connecting:

youtube-dl.exe -v "https://www.channel4.com/programmes/big-fat-quiz/on-demand/71755-001" => 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.channel4.com/programmes/big-fat-q
uiz/on-demand/71755-001']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2020.12.29
[debug] Python version 3.4.4 (CPython) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, phantomjs 2.1.1, rtmpdump 2.4

[debug] Proxy map: {}
[generic] 71755-001: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 71755-001: Downloading webpage
[generic] 71755-001: Extracting information
ERROR: Unsupported URL: https://www.channel4.com/programmes/big-fat-quiz/on-dema
nd/71755-001
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk
6o\build\youtube_dl\YoutubeDL.py", line 803, in wrapper
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk
6o\build\youtube_dl\YoutubeDL.py", line 824, in __extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk
6o\build\youtube_dl\extractor\common.py", line 532, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp581_vk
6o\build\youtube_dl\extractor\generic.py", line 3448, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.channel4.com/pro
grammes/big-fat-quiz/on-demand/71755-001

i.e., my error is

ERROR: Unsupported URL: https://www.channel4.com/programmes/big-fat-quiz/on-demand/71755-001

while yours is:

ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)'),))

superclarkk commented 3 years ago

@jesshub

Use yt-dlp fork and use param "--allow-unpla" Then you'll need to decrypt the content with key info.

Not supported with that either.

yt-dlp --allow-unplayable-formats -u <email address> https://www.channel4.com/programmes/celebrity-gogglebox/on-demand/72091-009

ERROR: [generic] Unsupported URL: https://www.channel4.com/programmes/celebrity-gogglebox/on-demand/72091-009

Vangelis66 commented 3 years ago

@superclarkk

As it's already known, Channel Four only offer DRMed VOD streams, to watch you have to

Be in a sanctioned location, i.e. the UK
Register an obligatory account with them and sign-in...
Use a sanctioned desktop browser with the latest version of the Widevine CDM (content-decryption-module)

DRM content isn't supported by downloading apps (at least not in the open 😜 ), for legal reasons; especially since the current owner of the most popular DRM solution (cenc+WidevineCDM) is the almighty Google; yt-dl's latest takedown by GitHub last year was "supposedly" DRM-related, so this is a strictly no-no topic...

yt-dlp doesn't directly support downloads from Ch4 either, so the -u option you issued in your posted command is also out of place...

As @jesshub advised, once you initiate successful playback in your desktop browser, you'll have to resort to URL-sniffing in that browser and unearth the direct URI to the actual MPEG-DASH MPD (media presentation description) file (i.e. a URI that contains .mpd); then, that URI has to be fed directly to yt-dlp along with the --allow-unplayable-formats flag... But yt-dlp doesn't support decryption of DRM-encrypted files, at best, up-to-this-stage, you'll end up with an encrypted/unplayable MP4 file... 😭

@jesshub wrote:

Then you'll need to decrypt the content with key info.

This isn't a job for the fainthearted and I doubt that it's even possible currently... I am aware that some months ago a Chromium extension (that won't be named) was able to extract decryption keys from previous Widevine CDMs, but last thing I heard was that Google were quick to patch that hole and, along the way, deprecate (on May 31st) all Widevine versions older than 4_10_2209_0; do you know otherwise (a simpe yes/no would suffice...) ?

Vangelis66 commented 3 years ago

but it works for channel 4 still

Thanks 😃 ; would that mean that older WidevineCDM versions still work to decrypt & deliver ch4 content in a browser?

Vangelis66 commented 2 years ago

I was able to successfully download and decrypt this media:

Just my 2c here, but posting in this tracker a link to a method of circumventing full-blown DRM on copyrighted audiovisual material is simply asking for legal trouble... Considering, not so long ago, youtube-dl was DMCA'ed by GitHub under the claim it could break DRM (not actually true) ... :wink: @dirkf, as the current maintainer, what do you reckon? (FWIW, Zombo has blocked me for some petty reason of his :angry: , but I still have lots of faith in you... :smile: )

dirkf commented 2 years ago

In so far as there is a policy, it's this. What some other GH repo does (and ofc they are all accessible by 2 clicks from yt-dl) is not our problem.

ytdl-org / youtube-dl

Support for channel4.com OnDemand #2050