TLV packet (Japanese advanced BS broadcasting)

[MMT-TLVassociation]

I'm using DumpTS to analyze packets of TLV/MMT stream captured from TV broadcasting by Japanese 4K broadcasting satellite. The problem I noticed is, when "DumpTS 4kmmt.mmts --showpack", DumpTS stops at 14th packet by error "[MMT/TLV] TLV header should start with 0x7F at file position: %"

4kmmt.mmts (captured MMT/TLV stream) is here, https://mega.nz/#!Rk4ijArA!xryyLP3Joo9vFJGMeHc_W323FLf_Zo0XwUL8mjPWdxg

I've checked the problem by VS2019 debug mode, then found "bs.curbits" is jumping to much far forward bit's position. The value is 0x81b1fed0.

Can you look at this problem?

[wangf1978]

Thanks for information, I have downloaded it, and will have a look.

[MMT-TLVassociation]

Thank you. I made a quick fix to line2245 of MMT.h back.data_unit_length = left_payload_data_len; //bs.GetWord();

It's working good now.

[MMT-TLVassociation]

Another problem I noticed is, mis-reading the length of TLV packet carrying "message" (MH-EIT etc) At line1962 of MMT.h length = bs.GetWord(); //bs.GetDWord();

A sample TLV stream carrying "message" is here, https://mega.nz/#!o8whgI5T!1_GG_KMQAKdOCxepxLJcuzAWAyLoiwpsUGQDcOZ4niI

Everyone can access the technical standards of MMT/TLV stream at the following link, https://www.arib.or.jp/english/std_tr/broadcasting/sb_ej.html

PDFs especially for the data structure of MMT/TLV are, http://www.arib.or.jp/english/html/overview/doc/6-STD-B32v3_11-3p3-E1.pdf http://www.arib.or.jp/english/html/overview/doc/6-STD-B60v1_13-E1.pdf

[wangf1978]

Thanks for your contribution, yes, I can get the spec, I will modify it.

[wangf1978]

I refer to 6-STD-B60v1_12-E1.pd.pdf and the PDF you specified, line2245 of MMT.h length = bs.GetWord(); I think it is according to the spec, please see Table 6-1 Configuration of MMTP payload In this table, it says data_unit_length is a WORD from stream

[wangf1978]

BTW: As for the message length, in some case, its length is 32bit, for example PA_Message (Table 7-1 Configuration of PA message), I am afraid we need process this case according to message_id.

[wangf1978]

As for line1962 of MMT.h, I suggest to change the code to

if (message_id == 0x8000 || message_id == 0x8001 || message_id == 0x8002)
    length = bs.GetWord();
else
    length = bs.GetDWord();

I confirmed it works with epg.mmts you provided

[wangf1978]

As for the code in line2245, I think the original code seems to be right according to 6-STD-B60v1_13-E1.pdf, in the spec, it says data_unit_length should be got from stream which used 16bits, is it possible the stream you provided is incompatible?

[MMT-TLVassociation]

Yes, please keep the original code. The TLV stream I have is captured from the 4K satellite broadcasting, so it's likely the stream is correct.

The problem is, if DumpTS uses data_unit_length value in reading MFU data bytes, its' size overs Payload_length (in line2190).

I think another quick fix for it is, (same as line2226) int actual_payload_size = (int)AMP_MIN((uint64_t)left_payload_data_len, (left_bits >> 3)); //int actual_payload_size = (int)AMP_MIN((uint64_t)back.data_unit_length - 14, (left_bits >> 3));

data_unit_length seems to represent a total bytes of MFU data (aggregation of MFUs) as shown in Figure 6-3 (a) in 6-STD-B60v1_13-E1.pdf so it's not a size of MFU data in the current MMT/TLV packet.

[MMT-TLVassociation]

For line1962 of MMT.h, The fix is perfect. Thank you.

[wangf1978]

data_unit_length (MFU length): This represents the size by the unit of Byte from immediately after this field to the last of one MFU data even according to 6-3 (a) in 6-STD-B60v1_13-E1.pdf, 2 MFUs shared the same header, and then data_unit_length is still the data unit size in

for (i=0; i<N; i++) {
    data_unit_length
    movie_fragment_sequence_number
    sample_number
    offset
    priority
    dependency_counter
    for (j=0; j<M; j++) {
        MFU_data_byte
    }
}

But I want to make a defensive fix for it

[wangf1978]

Just push a fix for these 2 issues

[MMT-TLVassociation]

Thank you. It's working very good. Option "--showinfo"for 4kmmt.mmts now shows "video_resolution: 7(0X7), height: 4320" It means this TLV stream contains 8K stream (4320P, 7680×4320 resolution).

For trollcop's sample.mmts, it's "video_resolution: 6(0X6), height: 2160" means 4K stream(2160P, 3840×2160 resolution). https://en.wikipedia.org/wiki/4K_resolution

Thanks again.

[MMT-TLVassociation]

I just noticed a little more fix needed for MFU data in case aggregation flag is 1. Please check the mmts file below, https://mega.nz/#!kx4ySKDQ!-oBSdoC3885LX8MugvyNF6QQWl-xBGDDOP3HcgOnOTY (It's a small part of TLV stream below) https://mega.nz/#!UwJkAS6J!pIjgt8mBFc4gG7nfE2SaKdaF4eil8fuz32Tj9Dscssw

The problem is , because back.data_unit_length is smaller than 14, DumpTS stops processing at this packet. I still don't know why the back.data_unit_length is such a small value ("5" this case). I just made a quick fix to skip this packet. from Line2255-2274 in MMT.h

if (left_bits < ((uint64_t)back.data_unit_length << 3) + 16)
return RET_CODE_BOX_TOO_SMALL;

left_bits -= 16ULL << 3;
left_payload_data_len -= 16;

if (left_payload_data_len > 0 && left_bits > 0)
{
int actual_payload_size = 0;
    if (back.data_unit_length < 14)
    {
    actual_payload_size = 0;//Just skip this MFU data bytes
    }
    else
    {
    actual_payload_size = (int)AMP_MIN((uint64_t)back.data_unit_length - 14, (left_bits >> 3));
    }
// Make a defensive fix here to avoid the data_unit_length exceed the left_payload_data_len, and    cause out of range of reading data
actual_payload_size = (int)AMP_MIN(left_payload_data_len, actual_payload_size);
if (actual_payload_size > 0)
{
back.MFU_data_bytes.resize(actual_payload_size);
bs.Read(&back.MFU_data_bytes[0], actual_payload_size);
left_bits -= (uint64_t)actual_payload_size << 3;
left_payload_data_len -= actual_payload_size;
}
}

[wangf1978]

Actually from the spec, I can't understand it very well, actually I think it broke the spec. I am not an expert for this part, if it works in your case, I think you can pull your request.

[MMT-TLVassociation]

Thank you. It's just a quick fix, but I could watch the generated *.hevc file normally, after a header conversion by dt2nal (it converts to NAL unit) dt2nal (for video data) http://codepad.org/uh7LvKJx

dt2aac(for audio data) http://codepad.org/jZIhb1NJ

[wangf1978]

Thanks for your contribution, merged.

[MMT-TLVassociation]

As for your comment on Dec 28, 2019, The data is correct as explained, but it's encrypted for some streams (paid channels or for rights protection)

data_unit_length (MFU length): This represents the size by the unit of Byte from immediately after this field to the last of one MFU data

for (i=0; i<N; i++) { data_unit_length movie_fragment_sequence_number sample_number offset priority dependency_counter for (j=0; j<M; j++) { MFU_data_byte } }

I'm modifying the code to extract M2 section message which contains keys to decrypt the headers (data_unit_length, etc.)

[MMT-TLVassociation]

https://github.com/MMT-TLVassociation/DumpTS/commit/b9a2d2f2777622a5217dc3be7c8a54eb863df1ed#diff-0ea5c26427da85cf198ce147b16bb283 The code is modified to extract decryption keys. It's temporarily output to three files. I still need more update.

[wangf1978]

Thanks for your contribution, do you need push the commit request here?

[MMT-TLVassociation]

Maybe not commit here, because the changes are mainly for decryption of TLV/MMT stream. The decryption needs an access to smart card, specially designed for TV broadcasting. I'm still modifying codes, but it's not completed.

https://github.com/MMT-TLVassociation/DumpTS/commit/72e8661b443df7e471356f40208b9c2ec2799cee#diff-25d902c24283ab8cfbac54dfa101ad31