UPID parsing with strange output

[wabiloo]

I'm using threefive in a tool that monitors a live HLS stream. The stream follows the "Addressable TV" spec (https://www.snptv.org/wp-content/uploads/2020/08/SNPTV-AFMM-Addressable-TV-UK-version-2.0.6.1.pdf)

It contains SCTE35 payloads with a 0x02 descriptor "CallAdServer", which contains a 16-byte upid of type "MPU" (Managed Private UPID - 0x0C), which should have the following structure:

• Bytes from 1 to 4: ASCII code for ‘ADFR’ = 0x41444652 • Byte #5: version number: by default 1 = 0x01 • Bytes 6 to 7: TV channel ID - CNI: example for TF1 = 0x33F1 • Bytes 8 to 11 (4 bytes): YYYYMMDD = 0x01341403 (20190211 in decimal) • Bytes 12 and 13: Ad break code: example = 0x0462 (break 1122) • Bytes 14 to 16: duration of the break in ms = 0x01C070 (duration: 114800 ms, meaning: 1minute, 54secondes and 20 frames).

Here is an example: 0xFC305E00014ECF5F9800FFF00506FE156DE4F0004802144355454900065E087FFF00001B77400000300710021F4355454900065EFF7FBF0C10414446520133F10134B04F065E060220020000020F4355454900065E077FBF00003106106F4BCB35

I'm a bit puzzled about the output of threefive for that UPID:

{
            "tag": 2,
            "descriptor_length": 31,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "components": [],
            "segmentation_event_id": "0x65eff",
            "segmentation_event_cancel_indicator": false,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": true,
            "segmentation_upid_type": 12,
            "segmentation_upid_type_name": "MPU",
            "segmentation_upid_length": 16,
            "segmentation_upid": {
                "format_identifier": 1094993490,
                "private_data": "0x133f10134b04f065e060220"
            },
            "segmentation_type_id": 2
        },

The private_data element does not seem to be a valid hex string. Or at least Python's bytes.fromhex() refuses it.

By the way, would you consider adding the "CallAdServer" segmentation type ID in your table22? In my code I simply add it in the following way: threefive.segmentation.table22[2] = "Call Ad Server"

[futzu]

I'm using threefive in a tool that monitors a live HLS stream. Check out showcues, for monitoriing live hls, you'll dig it.

It's valid hexidecimal, just not valid for byte conversion. there's an extra zero, not sure if it's my fault or your fault yet.

>>>> bytes.fromhex('133f10134b04f065e06022')

b'\x13?\x10\x13K\x04\xf0e\xe0`"'

• Format identifier
• I can update threefive to do the ascii conversion, is that what you want?

 int.to_bytes(1094993490,4,byteorder="big")
b'ADFR'

• The SCTE-35 Specification

• threefive code

    def _decode_mpu(self):
        mpu_data = {
            "format_identifier": self.bitbin.as_int(32),
            "private_data": self.bitbin.as_hex(self.bit_length - 32),
        }
        return mpu_data

`The stream follows the "Addressable TV" spec'

This is SCTE-35, I cannot commit to multiple specifications,.

None of this is SCTE-35,, but I made you a special threefive cli tool that does this , see below.

• Byte  version number: by default 1 = 0x01
• Bytes 6 to 7: TV channel ID - CNI: example for TF1 = 0x33F1
• Bytes 8 to 11 (4 bytes): YYYYMMDD = 0x01341403 (20190211 in decimal)
• Bytes 12 and 13: Ad break code: example = 0x0462 (break 1122)
• Bytes 14 to 16: duration of the break in ms = 0x01C070 (duration: 114800 ms, meaning: 1minute,
54secondes and 20 frames).

[futzu]

1) gunzip threefiveadfr.gz 2) install threefiveadfr ~/.local/bin

3) threefiveadfr '0xFC305E00014ECF5F9800FFF00506FE156DE4F0004802144355454900065E087FFF00001B77400000300710021F4355454900065EFF7FBF0C10414446520133F10134B04F065E060220020000020F4355454900065E077FBF00003106106F4BCB35'

• output

{
    "info_section": {
        "table_id": "0xfc",
        "section_syntax_indicator": false,
        "private": false,
        "sap_type": "0x3",
        "sap_details": "No Sap Type",
        "section_length": 94,
        "protocol_version": 0,
        "encrypted_packet": false,
        "encryption_algorithm": 0,
        "pts_adjustment_ticks": 5617180568,
        "pts_adjustment": 62413.117422,
        "cw_index": "0x0",
        "tier": "0xfff",
        "splice_command_length": 5,
        "splice_command_type": 6,
        "descriptor_loop_length": 72,
        "crc": "0x6f4bcb35"
    },
    "command": {
        "command_length": 5,
        "command_type": 6,
        "name": "Time Signal",
        "time_specified_flag": true,
        "pts_time": 3994.706311,
        "pts_time_ticks": 359523568
    },
    "descriptors": [
        {
            "tag": 2,
            "descriptor_length": 20,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "components": [],
            "segmentation_event_id": "0x65e08",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": true,
            "delivery_not_restricted_flag": true,
            "segmentation_duration": 20.0,
            "segmentation_duration_ticks": 1800000,
            "segmentation_message": "Provider Advertisement Start",
            "segmentation_upid_type": 0,
            "segmentation_upid_type_name": "No UPID",
            "segmentation_upid_length": 0,
            "segmentation_type_id": 48,
            "segment_num": 7,
            "segments_expected": 16,
            "sub_segment_num": 2,
            "sub_segments_expected": 31
        },
        {
            "tag": 2,
            "descriptor_length": 31,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "components": [],
            "segmentation_event_id": "0x65eff",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": true,
            "segmentation_upid_type": 12,
            "segmentation_upid_type_name": "MPU",
            "segmentation_upid_length": 16,
            "segmentation_upid": {
                "format_identifier": "ADFR",
                "version": 1,                          <-- Boom goes the dynamite
                "ChannelID": 13297,
                "YYYMMDD": "0x134b04f",
                "Ad Code": "0x65e",
                "Duration": 393760
            },
            "segmentation_type_id": 2
        },
        {
            "tag": 2,
            "descriptor_length": 15,
            "name": "Segmentation Descriptor",
            "identifier": "CUEI",
            "components": [],
            "segmentation_event_id": "0x65e07",
            "segmentation_event_cancel_indicator": false,
            "segmentation_event_id_compliance_indicator": true,
            "program_segmentation_flag": true,
            "segmentation_duration_flag": false,
            "delivery_not_restricted_flag": true,
            "segmentation_message": "Provider Advertisement End",
            "segmentation_upid_type": 0,
            "segmentation_upid_type_name": "No UPID",
            "segmentation_upid_length": 0,
            "segmentation_type_id": 49,
            "segment_num": 6,
            "segments_expected": 16
        }
    ]
}

threefiveadfr.gz

[futzu]

• threefiveadfr

import sys import threefive

def adfr_mpu(self): mpu_data = { "format_identifier": self.bitbin.as_charset(32), "version": self.bitbin.as_int(8),
"ChannelID": self.bitbin.as_int(16), "YYYMMDD": self.bitbin.as_hex(32), "Ad Code": self.bitbin.as_hex(16), "Duration": self.bitbin.as_int(24), } return mpu_data

if name == "main":

threefive.upids.UpidDecoder._decode_mpu = adfr_mpu

if sys.argv and sys.argv[1].lower() in [b'version','version']:
    print(f'{threefive.version}')
    sys.exit()
if len(sys.argv) > 1:
    if sys.argv[1].lower() in [b'pts','pts']:
        strm = threefive.Stream(sys.argv[2])
        strm.show_pts()
        sys.exit()
    if sys.argv[1].lower() in [b'show','show']:
        strm = threefive.Stream(sys.argv[2])
        strm.show()
        sys.exit()
    for arg in sys.argv[1:]:
        threefive.decode(arg)
else:
    threefive.decode(sys.stdin.buffer)

[futzu]

Update: we're both stupid .

It's just needs a leading zero. hex should always be an even number, if we want to convert to bytes.

bytes.fromhex("0133f10134b04f065e060220")
b'\x013\xf1\x014\xb0O\x06^\x06\x02 '

>>>> len("133f10134b04f065e060220")
23
>>>> len("0133f10134b04f065e060220")
24

>>>> bytes.fromhex("01" )
b'\x01'
>>>> bytes.fromhex( "33f1")
b'3\xf1'
>>>> bytes.fromhex("0134b04f")
b'\x014\xb0O'
>>>> bytes.fromhex("065e")
b'\x06^'
>>>> bytes.fromhex("060220")
b'\x06\x02 '

[futzu]

I built it into 2.4.25, you don't need threefiveadfr I left private_data in there, you need set that if you are encoding, it will ignore version and such.

    def _decode_mpu(self):
        mpu_data = {
            "format_identifier": self.bitbin.as_charset(32),
            "private_data": self.bitbin.as_hex(self.bit_length - 32),
        }
        if mpu_data["format_identifier"] =="ADFR":
            data = bytes.fromhex(mpu_data["private_data"][2:])
            mpu_data["version"] = data[0]
            mpu_data["channel_identifier"]= hex(int.from_bytes(data[1:3],byteorder="big"))
            mpu_data["date"] = int.from_bytes(data[3:7],byteorder="big")
            mpu_data["break_code"]= int.from_bytes(data[7:9],byteorder="big")
            mpu_data["duration"] =hex(int.from_bytes(data[9:11], byteorder="big"))
        return mpu_data

addressable TV compatibility

           "tag": 2,
          "descriptor_length": 31,
          "name": "Segmentation Descriptor",
          "identifier": "CUEI",
          "components": [],
          "segmentation_event_id": "0x065eff",
          "segmentation_event_cancel_indicator": false,
          "segmentation_event_id_compliance_indicator": true,
          "program_segmentation_flag": true,
          "segmentation_duration_flag": false,
          "delivery_not_restricted_flag": true,
          "segmentation_message": "Call Ad Server",   < --- Boom
          "segmentation_upid_type": 12,
          "segmentation_upid_type_name": "MPU",
          "segmentation_upid_length": 16,
          "segmentation_upid": {
              "format_identifier": "ADFR",  <--- Boom
              "private_data": "0x0133f10134b04f065e060220",
              "version": 1,                            <---- Boom
              "channel_identifier": "0x33f1",                  <---- Boom
              "date": 20230223,                         <---- Boom
              "break_code": 1630,                       <---- Boom
              "duration": "0x602"                <---- Boom
          },
          "segmentation_type_id": 2,         <----  Boom
          "segment_num": 0,
          "segments_expected": 0
      },

[wabiloo]

Awesome, thanks!