Channel 4 site changes - download fails

[GoogleCodeExporter]

What steps will reproduce the problem?
1. get_flash_videos http://....../4od#nnnnnn
2.
3.

What is the expected output? What do you see instead?
rtmpdump starting download...

What version of the product are you using? On what operating system?
Latest, OpenBSD current amd64

Please provide any additional information below.
The swf file has changed from 11.21.2 to 11.23.4, some programmes still work, 
depending on the streamURL returned by ais.channel4.com in the xml. Rather than 
a url ending with rtmpe://......mp4, it returns with http://.....fm4, a http 
query returns xml data, this now appear to allow additional media bit rate 
selection.

Original issue reported on code.google.com by njtaylor...@gmail.com on 13 Apr 2012 at 9:19

[GoogleCodeExporter]

Comment 47,48,49 - slist issue I managed to reproduce this. Works if stream URL 
is ak, but fails if ll the xml returned depends on the stream url selected. 
Updated perl script should work for both ak and ll

Comment 50 - I don't get that problem, did a compare of outputs, and contents 
of 1st Fragment must differ - all the same up to where your goes wrong. Could 
your file have been truncated?

$ sha1 ch4_3412042/*Frag[0-9]
SHA1 (ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag1) = 
02bad191229a9da9f4e59f77f20dd671d3aae588
SHA1 (ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag2) = 
0f3b228ad1a045a56326288d710500d1ef1252ca
SHA1 (ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag3) = 
9ea0d82d52a21f3945ca56162a64a57479e74c04
SHA1 (ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag4) = 
c638b8800165563d679afe8a024a45b223a435e5

$ ls -l ch4_3412042/*Frag[0-9]
-rw-r--r--  1 ntayl01  ntayl01  706542 Nov  8 11:45 
ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag1
-rw-r--r--  1 ntayl01  ntayl01  708510 Nov  8 11:45 
ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag2
-rw-r--r--  1 ntayl01  ntayl01  721886 Nov  8 11:45 
ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag3
-rw-r--r--  1 ntayl01  ntayl01  756449 Nov  8 11:45 
ch4_3412042/CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag4

These are large to process in memory, could be some limitation of version of 
perl used. What OS/Perl are you using.

Original comment by njtaylor...@gmail.com on 8 Nov 2012 at 12:22

Attachments:

• ch4_f4v_flv.pl

[GoogleCodeExporter]

i am using activeperl 5.14.2 on windows 7, i have tried on my linux machine but 
i get blowfish module type things error not sure how to install the modules on 
linux.

i will give the lastest version a try

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 12:37

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

If anyone has successfully downloaded entire video file, even though it doesn't 
play, can you upload it to some http location so that I can run it through the 
Adobe DRM test app?

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 4:27

[GoogleCodeExporter]

There's a good chance there's no key in any of the requests. HTTP response 200 
can be enough to give a green light to the player to start playback with a 
192-byte DRM token that's a part of asset descriptor: 
http://ais.channel4.com/asset/3420568 (uriData->token).

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 4:40

[GoogleCodeExporter]

i can attach a  few files but not the full lot because there over 1000

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 5:47

Attachments:

• CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag1
• CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag2
• CH4_08_02_24_45569001001003_003_16x9_1500000_Seg1-Frag3

[GoogleCodeExporter]

I tried separate files but none of them are seen by DRM tester as video files. 
Here is the link: http://drmtest2.adobe.com/AccessPlayer/player.html
Paste this url as an example: http://onlinelib.de/tmp/output1.mp4 and you will 
see what happens when a generic Flash/Flex video component sees a DRM protected 
file. 
In case of our segments they are not seen as protected video, or any kind of 
video as a matter of fact.

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 6:00

[GoogleCodeExporter]

that sites basically tells you how the encyption decyption works

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 6:18

[GoogleCodeExporter]

How DRM routines work, yes. And it is supposed to do the same thing with 4oD 
files.

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 6:27

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

there more than just login method but i do find it strange it says no drm, but 
i suspect the files itself isnt protected but the transmission is

so 4od sends teh file in fragments and each is encypted on the fly so unless 
you have the decryption key it is meanaless

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 6:34

[GoogleCodeExporter]

Comment 60: Exactly! This is exactly what I have asked for in comment 54.

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 6:36

[GoogleCodeExporter]

comment 62 the problem is we cant combine the files into a file but i tried teh 
direct link of the file and it says no drm

interesting

DRM Error: 3304[AuthorizationFailed] 
Load 
http://ll.abrstream.channel4.com/CH4_08_02_24_45569001001003_003_16x9_1500000_.f
4m 
Stream not found 

it says no drm but wheni try ot play it it says yes drm so it seems only when 
the stream is called for that it is encypdted confirm there isa  key somewhere

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 6:42

[GoogleCodeExporter]

Comment 63: that's ok, this happens because drmtest2.adobe.com hasn't pinged 
the licence server like our client machines do when flash player initializes 
playback.

In AdditionalHeader there's a attribute EncryptionAlgorithm = AES-CBC with 
encrypted key length. So comment 61 saying the traffic is probably encrypted 
may be right. There are ways to encrypt traffic like this: 
http://bsdsupport.org/2007/03/q-how-do-i-encrypt-file-transfers-with-dd-and-netc
at/

Original comment by aulisme...@gmail.com on 8 Nov 2012 at 6:57

[GoogleCodeExporter]

comment 64 i was comment 61 i understand the process and i know about the 
aes-cbc enycption i dug it out of the drm string :)

the question is where is the key and hwo is it transmit fromt eh client to the 
server the content itself isnt encypted jdut teh transmission

Original comment by andrewcr...@gmail.com on 8 Nov 2012 at 7:01

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

yip that is it, i am goign to need ot rea it but if my quick read was anything 
to go by they think by removing the client to license server ie ther eno 
licesne server that ther eno man in the middle but the informaiton to get the 
decrytion is on that page but we are missing some data which we nee dto ifnd

sorry my spelling is worse than normal bit tired on top of ym dsylexica

Original comment by andrewcr...@gmail.com on 9 Nov 2012 at 7:42

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

If this is based on SWF verification for Protected HTTP Dynamic Streaming 
(http://tinyurl.com/cpvqjo4), then each received segment of video data has some 
sort of a header (with whitelisted swf clients and maybe some other stuff), 
real FLV header metadata is in corresponding f4m, and .... video data in each 
segment is probably unprotected, and so is the transport? There should be FLV 
keyframes in each segment. Can anybody help parsing raw segments completely 
disregarding the header?

Original comment by aulisme...@gmail.com on 9 Nov 2012 at 9:54

[GoogleCodeExporter]

Comment 66, Had a look, says either PHDS or FlashAccessV2 may be selected - 
from f4m decoding so far we have

Encryption Metadata Encryption Version 2 SubType = FlashAccessv2 Method = 
Standard Algorithm AES-CBC Key Length 16

Suggests it's not PHDS, but Flash Access V2

Looking around this was interesting...

http://www.adobe.com/devnet/adobe-media-server/articles/dynamic-streaming-protec
tion.html#content_encryption

common key (128bytes) + content id create the encryption key.

Have this...
DRM ID = drmMetadata5154 DRM Content ID = 293409

http://help.adobe.com/en_US/HTTPStreaming/1.0/Using/WSaeac10ab694095a12a9a3a7d12
823cda643-7ffc.html#WS52e69fdca9fe1cf2-563168fd12623f1bd2d-8000

--common_key only first 16bytes are used to generate the key

Elsewhere,

The Content ID to be used for content protection. If not specified, the salt is 
the filename. If specified, the salt is shared with all content in the 
directory.

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 12:10

[GoogleCodeExporter]

Reply to comment 69.    "Can anybody help parsing raw segments completely 
disregarding the header?"

The perl script processes each segment, the Video+Audio in encrypted in the 
segments. This is the extracted Video/Audio stream which the script displays...

Durat  Size   Flags     COffs Type Leng  Timesta StrID Aud/VidHdr filt Name 
Filtlen EFlag IV                                PLen TSize
    0,    57, 02000000,     0 28      42       0     0 af00          1 SE        17    80 9640951c3b03a68f0471d8a6e7741736    16    53
94a9175d4d31ca851a3d9eb43a17e130
6fdbc1d5333b8c77fdb713b25a6fa8e4
    0,   124, 02000000,     0 29     109       0     0 1700000000    1 SE        17    80 fe7b636b96154b0f62fca05abec5fa75    80   120
7970a267decd081001d3a59a58f7225435b430589ad5e92aae3e48c0f4ba1ff52199cce12c2bc7d7
97d867e9d8d2c586
7ea3f233649517482ff5502fe0e9f97a04ff200fb63a4e5b1ffd0ed3b12d65e0a753db99a3b026a9
be667a4a65806481
    0,   124, 02000000,     0 29     109       0     0 1700000000    1 SE        17    80 10ff85f3e8357fc666fb5fa2129509a6    80   120
5c0d7ad8f4d5bdd6015fb575fbdbe4adcee851ba4464e21dd5d5bcefd6a7777b860138e9f8af0609
875ca2dd5358e0c3
616634f9ae791a7c8d5fdec3b7e057f70ac4bf4dd025db7f38a915d6f1796abe917299478ce8ea2d
1524970a222549d4

Column Flags/Type - 0x08/09 = Audio/Video 0x28/9 = Encrypted Audio/Video.
Column filt = Pre-processing filter used.
Column Name = Pre-processing filter name SE = Standard Encryption.
Column Filterlen = 17 bytes = E Flag + IV
Column IV = Initialization Vector - used with AES encryption/decryption.

The two hex strings are the video/audio before + after decryption.

The messages Padding error which occurs, is because the data is padded to a 
multiple of 16bytes, 
See padding for PKCS7 is described in RFC 5652. If adding 5 characters, then 
the padding character is 5 repeated 5 times.

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 1:00

[GoogleCodeExporter]

i think i can work out the key from reading the documents, njtaylor can you add 
complex maths and string calculations, and string manipulation with perl? if so 
i can try work out the pesudo code for it and you can try fit it into the script

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 2:28

[GoogleCodeExporter]

Yes, if you can work it out, I will be able to convert from pseudo code to 
perl. 

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 3:08

[GoogleCodeExporter]

what show is that above segments from i will need to use that as my example. is 
the top hex string the before and the bottom one after?, i assume the top one 
is the encrypted value? and the bottom is using base value for decyption but it 
is wrong?

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 6:03

[GoogleCodeExporter]

Yes
IV = 9640951c3b03a68f0471d8a6e7741736

encrypted = 94a9175d4d31ca851a3d9eb43a17e130
decrypted = 218890f98d372aea06d129f20540f543 (with'n9cLieYkqwzNCqvi' as key) or 
6fdbc1d5333b8c77fdb713b25a6fa8e4 (in comment 71 was a different key)

The strings are truncated to the first 96 characters in the output.

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 6:24

[GoogleCodeExporter]

what show was that grabbed from? i need teh manifest, xml, der, swf with that 
show and episodes that way i can work it out or at least try to

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 7:15

[GoogleCodeExporter]

This 

 http://www.channel4.com/programmes/time-team-specials/4od#3412042

attached files.

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 8:04

Attachments:

• ch4_3412042.der
• asset.xml
• 3412042.html

[GoogleCodeExporter]

cheers i will start looking at it tonight going to take a while to get some 
information

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 8:09

[GoogleCodeExporter]

whilst looking for the key information i have noticed something in the der 
file, a time stamp of when the file was accessed, seems there logging :)

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 10:02

[GoogleCodeExporter]

can you change it to download about 800 fragments or all 1200 i need to see if 
the 80 flag is on them all or not

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 10:38

[GoogleCodeExporter]

this seems like it is the common key

00 DD C9 5D 9B 84 87 D5 14 46 26 45 8E 79 02 80 
7C 80 B6 C8 5B 26 77 47 13 1D 2F 50 2A 7A 52 B2 
D0 34 8A B6 E5 39 E2 39 E9 6B 9F AB 35 E8 0F F3 
CA 7C 71 73 F6 15 A3 36 DF 4A 85 74 61 BB 8E 92 
08 EC 82 A9 12 38 0C A6 1D F3 5C AE 9A 1E 96 9D 
3F 7B 62 D9 DB 1D 62 61 72 2B A2 1F B9 C0 A3 9D 
02 95 23 68 6C 15 C8 FD 19 97 7A B2 52 EA D6 41 
5E 83 92 FE 93 A3 89 D1 36 C7 29 99 9C AC 30 0D 
F9 

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 10:51

[GoogleCodeExporter]

in sequence (1705,242) within the der file conents information relating to the 
file goign to be played, and (6111,128) has the invidual file key, the common 
key and the ivindual key are used to create teh decrytion key

there isa possiabilty this could be the common key as well 

00 B3 CB 19 23 5D 45 6A 49 87 64 78 9B AE 21 24 
02 21 87 85 98 13 8E 3C 94 E4 CE 51 47 DE 90 70 
70 F4 42 23 B1 79 8E 10 F0 50 25 26 AA 14 72 C3 
9D 72 56 FE 3B 24 B6 AE 7B 4F 54 8A 74 57 28 1D 
76 EE A7 49 2E 12 6A 27 10 72 13 CB 27 11 E3 4B 
F1 11 46 42 34 69 EA 6F 1D 5D 5E 97 A2 DD 70 35 
D0 FF D7 94 BD 09 D8 C3 AF 51 35 D6 F2 4B D0 87 
01 ED BB 18 C3 C1 8E 9E 19 2B 8C 9D 3D 3E 12 7A 
02 3A 25 B9 7C FC 09 E9 A4 02 1E AE 03 CB 2A 81 
98 63 1E 9A 7E D1 C0 89 46 C6 40 7A 88 1B 75 75 
46 36 6A E7 95 57 BD E7 92 06 77 4A 63 D9 9C D0 
EB 59 11 97 7E 3B 5B BD 40 A9 2D BB 5A B9 E1 79 
7E 4D 7B BA B3 12 B5 32 58 F8 73 7C 17 DF E8 ED 
98 9D FA F5 0E 4A DA 7A 77 BE 39 F8 12 A6 03 16 
5E D8 89 A0 E7 EE 8E 59 29 1D 03 EC 55 0C 3E E7 
20 EF 51 E6 93 63 EA 70 77 10 E1 FE 9C 95 5C 87 
1B 

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 11:00

[GoogleCodeExporter]

1705,242 might also contain some decyption information

Original comment by andrewcr...@gmail.com on 10 Nov 2012 at 11:02

[GoogleCodeExporter]

These lines, around line 746...

print "Fetching $numfrags fragments\n";
my $bar = Term::ProgressBar->new( { count => $numfrags, name => "Download 
Fragments " } );
print "Reduced fragments\n";
$numfrags = 4;

Change 4 in  $numfrags = 4; to whatever number, or put # in front to comment 
out then will download all segments.

Original comment by njtaylor...@gmail.com on 10 Nov 2012 at 11:24

[GoogleCodeExporter]

Checked with all fragements downloaded, and the flag 80 is set on all 219724 
audio/video samples.

Original comment by njtaylor...@gmail.com on 11 Nov 2012 at 1:10

[GoogleCodeExporter]

ok that means there using the default setting which is encrypt all fragments 
and not using option 1 or 0 which would only be 50% or 20%

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 10:11

[GoogleCodeExporter]

ok doing base64 calculatiosn and hex calculatiosn isnt easy but her is what i 
think the key should be

psuedo code

commonkey = 00 DD C9 5D 9B 84 87 D5 14 46 26 45 8E 79 02 80 
7C 80 B6 C8 5B 26 77 47 13 1D 2F 50 2A 7A 52 B2 
D0 34 8A B6 E5 39 E2 39 E9 6B 9F AB 35 E8 0F F3 
CA 7C 71 73 F6 15 A3 36 DF 4A 85 74 61 BB 8E 92 
08 EC 82 A9 12 38 0C A6 1D F3 5C AE 9A 1E 96 9D 
3F 7B 62 D9 DB 1D 62 61 72 2B A2 1F B9 C0 A3 9D 
02 95 23 68 6C 15 C8 FD 19 97 7A B2 52 EA D6 41 
5E 83 92 FE 93 A3 89 D1 36 C7 29 99 9C AC 30 0D 
F9 ( i will find where it is in the der file for you as the common key can 
change over time)

filekey = derfile squence (6111,128)
16byte commonkey = string(commonkey) 16 bytes
16byte filekey = string(filekey) 16 bytes

decyrtionkey = 16byte commonkey+filekey

not sure how the key is transmitted but by the looks of it you have managed to 
transmit it such it gives a different after hash key

if you can code that above into the script and rerun it for the first 4 
fragments and i can see how the hash value compares to the value you used before

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 2:05

[GoogleCodeExporter]

sorry meant compare the two hex strings before and after

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 2:06

[GoogleCodeExporter]

might also want to do some error checkling with this line

psuedo code

aes128bit check = stringlength(decyrtionkey)

print aes128bit check

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 2:08

[GoogleCodeExporter]

also we require the grabber to imtait the player so

player = flashplayer11.1 (not sure how to make it seem like the flash player 
but if it isnt then the server can ban certain versions)

this will have to be in the verication process note this option or black and 
whitelisting isnt be used yet on 4od

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 2:55

[GoogleCodeExporter]

ok seem the new swf file is the player

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 2:57

[GoogleCodeExporter]

for http://www.channel4.com/programmes/time-team-specials/4od#3412042
this might acutalyl be the common key

E885996397A4A2D635CE48780341359C0245F765

and this the content id 

323933343039

or using the original common key

00DDC95D9B8487D5144626458E7902807C80B6C85B267747131D2F502A7A52B2D0348AB6E539E239
E96B9FAB35E80FF3CA7C7173F615A336DF4A857461BB8E9208EC82A912380CA61DF35CAE9A1E969D
3F7B62D9DB1D6261722BA21FB9C0A39D029523686C15C8FD19977AB252EAD6415E8392FE93A389D1
36C729999CAC300DF9

add that and the content key together and youg et 128bit key so this could be 
the dercytion key as well, not easy to determine the exact values to be used as 
ther eisa few possible once i can work out from trail and error i can then 
decrypt it

but it will require stripping the der file of the information in the future as 
it could change but for now common values will be best

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 3:28

[GoogleCodeExporter]

The .der is extracted from the asset.xml, that is fetched using http, so
just a question of setting the user agent plus any html cookies /
parameters for the request. The agent routine allows setting the user
agent, or the agent can be specified in WWW::Mechanize->new(agent =>
'.....'). Just need to check the some captured HTTP requests and set
accordingly.

Original comment by njtaylor...@gmail.com on 11 Nov 2012 at 3:37

[GoogleCodeExporter]

cool post the tests results once you have scripted the attempt at the key, but 
the decytion key is without a doubt in the der file but there is quite a few 
possiabilties certainly the common key which i found in 3 places all different 
but the same on 5 der files has changed since yesterday but not greatly so th 
output of the segments headers will give me idea if i am close to the right 
information and the der files has two content spefic information and keys so i 
need ot look at them as well

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 3:55

[GoogleCodeExporter]

readiong through the documents tehre is a small possiabilty that 4od is using 
caching option, so basically the fragments are stored on your computer up to 
maximum of 24 hours, but that time can be changed so it could be there there 
decrypted for a few minutes, all luser watching on linux, mac and windows i 
suggest doigna full system scan for any of the programs on 4od using this 
method that you have been trying to rip, i also suggest checking common temp or 
cache location when trying to stream it live

Original comment by andrewcr...@gmail.com on 11 Nov 2012 at 9:03

[GoogleCodeExporter]

Sure, all the segments are in a browser cache, I saw them since day one. 

Original comment by aulisme...@gmail.com on 11 Nov 2012 at 10:57

[GoogleCodeExporter]

comment 96 try and use the adobehds.php to join them see if it reports it as 
encypted, if i have chance ill look for them myself and try

Original comment by andrewcr...@gmail.com on 12 Nov 2012 at 9:41

[GoogleCodeExporter]

it seems teh cached ones are encypted to need to see how nj script works with 
te keys

Original comment by andrewcr...@gmail.com on 12 Nov 2012 at 10:57

[GoogleCodeExporter]

Extracting keys... look for start point using OID 1.2.840.113549.1.1.1, 
should be followed by NULL (0x05) identifier, then either BIT (0x04) or Octet 
(0x03) String identifier wi..
Finds 5 keys, 3 128 bytes, 2 256 bytes 

Attached - revised script
extract keys - tries all five keys using first 16bytes. then first 16bytes of 
five key + ContentId.

Attached output from script...

Original comment by njtaylor...@gmail.com on 12 Nov 2012 at 2:38

Attachments:

• ch4_f4v_flv.pl
• xx.out

[GoogleCodeExporter]

ok ill have a look seems i am goign to need to try different combinations

Original comment by andrewcr...@gmail.com on 12 Nov 2012 at 3:52