Closed bbgdzxng1 closed 1 year ago
futzu
commented
1 year ago I made a fix for it several months ago, but never put it in, I think I just forgot about it. Ill try to get something working later today.
For part 2, just add quotes
#EXT-X-CUE-OUT-CONT:TEST1="helloWorld",TEST2="helloWorld2=",TEST3="helloWorld3==",TEST4="hello=World4",SCTE35="/DAvAAAAAAAA///wFAVIAACPf+/+c2nALv4AUsz1AAAAAAAKAAhDVUVJAAABNWLbowo=="
bbgdzxng1
commented
1 year ago As ever, thanks Adrian.
These vendor-specific tags are a pain. So many flavors, so much abuse of tags. I have never been able to trace which vendor started using EXT-OATCLS-SCTE35 messages (Unicorn/Brightcove? Verizon/UpLynk? Elemental?). I don't even know what the OATCLS acronym is supposed to stands for. But yeah, looks like it is common in the real world.
#EXT-OATCLS-SCTE35 in their samples.#EXT-OATCLS-SCTE35 in their samplesAs for #EXT-X-CUE-OUT-CONT, Amazon Elemental does not quote them in their samples either. Here's Amazon's example, which reflects the output of MediaLive.
#EXT-OATCLS-SCTE35:/DA0AAAAAAAAAAAABQb+ADAQ6QAeAhxDVUVJQAAAO3/PAAEUrEoICAAAAAAg+2UBNAAANvrtoQ==
#EXT-X-ASSET:CAID=0x0000000020FB6501
#EXT-X-CUE-OUT:201.467
.
.
.
#EXT-X-CUE-OUT-CONT:ElapsedTime=5.939,Duration=201.467,SCTE35=/DA0AAAA+…AAg+2UBNAAANvrtoQ==
.
.
.
#EXT-X-CUE-INOf course, you are absolutely correct regarding HLS - thanks for clarifying in your own inimitable style. Technically correct is the best kind of correct.
Assuming that your workaround for non-quoted tags works, could you add self._do_cue() for #EXT-X-CUE-OUT-CONT around line 331? I assume that's the part that triggers the decode to cue_data.
My use case is that I have a third-party source which has had one or many of:
EXT-X-CUE-OUT:[DURATION], #EXT-X-CUE-OUT-CONT:Duration=[DURATION] and the duration declared in the B64 SCTE payload. Nasty. A duration is communicated in three places for the same event, which is just foul.#EXT-X-CUE-OUT-CONT messages appearing for some time after the #EXT-X-CUE-IN, which can occur if an upstream source is restarted mid ad-break. Again, not clean.Manually debugging with good-ole threefive is a painfully manual process (but does work), so I looked to m3ufu to parse-and-decode the cue_data to rapidly identify which SCTE messages were rogue. Of course, that put me down a rabbit hole of cue_data not getting decoded.
If you have the patience to add support for unquoted, non-standard tags that do occur in the real world, that would be most welcome. However, I appreciate that any custom handlers may subtract from the purity and elegance of the statement in https://github.com/futzu/m3ufu/commit/faea00b7bda428bca1b60e37ae8a5c55baa2109a, which in itself is an honorable goal.
I hope all is well in Atlanta. Thanks again.
futzu
commented
1 year ago I can add something OAT specific, I am willing to do that for SCTE-35 stuff, but I can't currently think of a sane way to do it for the question 2 stuff. SCTE-35 base64 cues can have commas and equal signs in them, parsing other custom attributes with unquoted values is a recipe for disaster. I don't see a way to do it without having the attributes names.
forgive me, but I don't understand what you need to accomplish with the CUE-OUT-CONT tags, what is the goal? I understand the errors, but I am missing something here, I am a bit slow today.
bbgdzxng1
commented
1 year ago Purpose of the CONT messages? CUE-OUT-CONT tags are used by a session-based SSAI or DAI server when a stream is joined mid-break, so that the SSAI or DAI server know how much remains (the original CUE-OUT may have dropped out of the sliding window). HLS manifest CONTS are also used to update the length of the break in live sports for "Return to Studio", much like a true SCTE splice immediate - ie to communicate a change in duration prior to a CUE-IN. Please excuse the over-explaination.
So some SSAI/DAI servers will actually act on CONT messages, and treat them as "last known update" and thus they need to be validated during stream debug. Other less sophisticated SSAI/DAI servers just ignore the CONT messages and serve an empty break, if joined mid break. Other SSAI/DAI servers are stateful. [ And of course, it is accepted that CONT messages are a non-standard vendor implemented tag, and are unquoted. ]
As for my case...I have a stream from a third party source which is a bit of a mess... there are cues all over the place. The broadcaster's intention was one of figure 1 from the SCTE spec, but the reality of the stream does not match up with the figure below from the SCTE spec.
I don't have direct control over that stream, but I do want to help my third party debug their output. And it needs debugging, and that's why I turn to m3ufu.
The problem that there is contradictory info in the cues in the stream. So I was trying to use m3ufu to decode the cue_data for debugging purposes, ie identify which INs went with the OUTs, which spliceIds were paired etc. Throw on top of that a bunch of CONT message updates and debugging it manually is an exercise in copy-paste to threefive. Whereas with m3ufu, debugging becomes much easier.
So, I wanted to look at the cue_data in the [unquoted] CONTS to validate what was in each SCTE message.
Does that explain the background and my use case of m3ufu?
futzu
commented
1 year ago The CONT cue should be identical to the CUE-OUT cue, sounds like they have much deeper mpegts parsing issues, tell them to look at x9k3.
make a new directory Dm3ufu put this in that directory and rename it Dm3ufu.py Dm3ufu.py.txt
it will print everything a segment contains,
Segment: https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1673688358_video_720p-30fps-3436kbps/video_38167.ts
PTS Start from MPEGTS 24149.799067
_lines = ['#EXT-X-KEY:METHOD=AES-128,URI="https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/3a2f156105c692c9d930e5509ddd4761d7b2efdc.key",IV=0xF23AAF96F6DE7EC91047F6596238AE39', '#EXT-X-CUE-OUT-CONT:ElapsedTime=90.099,Duration=210.400,CAID=0x000000003DDD74D4', '#EXTINF:6.006,', '1673688358_video_720p-30fps-3436kbps/video_38167.ts']
media = https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1673688358_video_720p-30fps-3436kbps/video_38167.ts
pts = 24149.799067
start = 24149.799067
end = 24155.805067
duration = 6.006
cue = False
cue_data = None
tags = {'#EXT-X-KEY': {'IV': '0xF23AAF96F6DE7EC91047F6596238AE39', 'URI': 'https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/3a2f156105c692c9d930e5509ddd4761d7b2efdc.key', 'METHOD': 'AES-128'}, '#EXT-X-CUE-OUT-CONT': {'CAID': '0x000000003DDD74D4', 'Duration': 210.4, 'ElapsedTime': 90.099}, '#EXTINF': 6.006}
tmp = noaes-video_38167.ts
base_uri = https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/
Thankyou, sir. I have that installed and running.
I did have a think about the logic for the two cases.
#EXT-OATCLS-SCTE35, I think this tag is still within the HLS spec without the quotes, since there is no comma in the tag. Since there is no comma in the tag, then it is not technically an "attribute list" tag, and it is just a plain old un-enumerated string, which I believe is allowed. Of course, best practice would be for the encoder to quote them. But I think it is still valid.#EXT-OATCLS-SCTE35:/DA0AAAAAAAAAAAABQb+ADAQ6QAeAhxDVUVJQAAAO3/PAAEUrEoICAAAAAAg+2UBNAAANvrtoQ== EXT-X-CUE-OUT-CONT, as you say, this is certainly not compliant with HLS spec, as it does not comply with the definition of "https://www.rfc-editor.org/rfc/rfc8216.html#section-4.2", which states that all non-enumerated strings should be of type quoted-string.However, I was thinking that it must be parse-able, because SSAI/DAI servers are using it in the real world.
#EXT-X-CUE-OUT-CONT:ElapsedTime=5.939,Duration=201.467,SCTE35=/DA0AAAAAAAAAAAABQb+ADAQ6QAeAhxDVUVJQAAAO3/PAAEUrEoICAAAAAAg+2UBNAAANvrtoQ== Here's where I was thinking:
[A..Z], [0..9] and '-'.+ \ =.EXT-X-CUE-OUT-CONT is treated as the delimiter, after separation by commas, it should, in theory be robust.I agree that this not nice, because it is a workaround on a non-compliant HLS tag. But I think that cutting on commas before equals, and only treating the first occurrence of equals as a the delimiter between AttributeName=AttributeValue, should allow m3ufu to decode EXT-X-CUE-OUT-CONT. But I suspect that you're less than keen on supporting non-compliant tags.
As for Dm3ufu.py - HUGE thanks for sharing. I'm still eagerly awaiting ad break. I'll take a look at one of the OATCLS when my log capture has finished.
I really appreciate you, sir.
futzu
commented
1 year ago The reason I think you're having issues with your segmentation descriptors is that HLS only considers the first descriptor of a SCTE-35 cue, I forget the how and the why, you may want to look it up. I had an issue with that a while back.
bbgdzxng1
commented
1 year ago I really appreciate the Dm3ufu.py - thankyou for taking the time to tweak something for one of your loyal users. It didn't decode the OATCLS as I expected, but I'll look at it in more detail of the coming days and compare the output of m3ufu, Dm3ufu and threefive. I expect I'll be better off decoding each message manually with threefive.
I do appreciate your perspective. Part of the process of debugging is to make sure I have cross checked every message in the avail to identify which HLS cues are overlapping - I'll defend my case that I'm absolutely not going too deep - part of debugging is to go deep, identify each and every different cue message that covers the avail. A downstream SSAI/DAI server is having to act and decision on these messages - and to debug that, I have to start with decoding the source. Therefore I need to decode each and every message across the nested break to see which cues pair with which - whether that is a OUT/IN, a OAT, a CONT or a Base64 because they may have contradictory information, hangovers from previous avails etc. Debugging a complex nested stream felt like a genuinely reasonable use of m3ufu, because if every stream was perfect, we would not need analysis tools.
Unquoted SCTE messages do exist in the wild - I can't escape that (if you excuse the pun). I'm not in control of the original broadcast feed, I'm not in control of the encoders, nor in control of the code that defines the industry markup of the tags. If decoding unquoted SCTE messages is not for m3ufu, then I'll count my blessings and be grateful for your threefive and grep.
I do worry that I have upset or offended you. That was never my intent. I feel like through genuinely good intent to try to decode real-world markup from a mainstream encoder vendor has pushed the boundaries of your patience. I hope to continue using your tools for many years to come - and for that I would rather you don't consider me to be a pain in the ass. I'm just a loyal and enthusiastic user.
I feel really bad about suggesting that unquoted OATCLS and CONT could be useful to users. I'll close the ticket, because I've already taken up too much of your valuable (unpaid, voluntary but highly appreciated) time.
Thanks, and sorry.
futzu
commented
1 year ago I'm sorry if I was short with you man, I had to put my mom in a long term care facility yesterday, and I got a new project that requires me to get out bed before noon. On top of all that, I ran into a car on my brand new electric unicycle, looking like a damn fool. Bear with me, I am trying to get myself together.
futzu
commented
1 year ago I didnt forget about you though, oats.py.txt
I got the goofy tags parsing correctly, thats just a test.
futzu
commented
1 year ago a new test, I got to get back to my job, Ill work on it on it over the weekend oats.py.txt
futzu
commented
1 year ago I just about have it all figured out, and I think I can do it without having to make it a "special" situation.
In other news: Let me ask you this though, are the scte-35 cues embedded in the video? Something I have not really advertised is that you can use m3ufu and x9k3 to re-segment an m3u8 and add SCTE-35, or in your situation, repair it. and its fast enough to do it live.
futzu
commented
1 year ago I wasn't mad at you man, but I dont get paid for this so I don't want spend my time writing code to debug some companies shitty SCTE-35 software, which they should be fixing themselves.
That being said, I do like you and want to help you if I can.
>>>> from m3ufu import TagParser
>>>> tp = TagParser(["#EXT-OATCLS-SCTE35:/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g="])tp.tags {'#EXT-OATCLS-SCTE35': '/DAvAAAAAAAA///wBQb+dGKQoAAZAhdDVUVJSAAAjn+fCAgAAAAALKChijUCAKnMZ1g='}
have m3ufu reassemble the segments, this will also trigger writing insert times and cues to sidecar.txt
m3ufu -i bad.m3u8 -o bad.tsthen in another terminal, let x9k3 segment the reassembled video and insert the cues.
x9k3 -i bad.ts -s sidecar.txt --liveI tested that you can do it live, or vod, and it worked very well and was very fast.
futzu
commented
1 year ago You want all the details, check this out. m3ufu-debug.py.txt
{
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 70,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0x0",
"tier": "0x0",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 48,
"crc": "0xf604d2fb"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 13036.045733,
"pts_time_ticks": 1173244116
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 46,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x400dce0f",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": true,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": false,
"no_regional_blackout_flag": false,
"archive_allowed_flag": false,
"device_restrictions": "Restrict Group 0",
"segmentation_duration": 255.233556,
"segmentation_duration_ticks": 22971020,
"segmentation_message": "Provider Placement Opportunity Start",
"segmentation_upid_type": 13,
"segmentation_upid_type_name": "MID",
"segmentation_upid_length": 24,
"segmentation_upid": [
{
"upid_type": 10,
"upid_type_name": "EIDR",
"upid_length": 12,
"segmentation_upid": "10.5239/2435-a45b-0-0-0"
},
{
"upid_type": 8,
"upid_type_name": "AiringID",
"upid_length": 8,
"segmentation_upid": "0x3df78df3"
}
],
"segmentation_type_id": 52,
"segment_num": 0,
"segments_expected": 0,
"sub_segment_num": 0,
"sub_segments_expected": 0
}
]
}
Media: https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts
Lines Read: ['#EXT-X-KEY:METHOD=AES-128,URI="https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",IV=0xBDFAEF4BF68E82BB3CC3C576E6C11117', '#EXT-X-ASSET:CAID=0x000000003DF78DF3', '#EXT-X-CUE-OUT:255.233', '#EXT-X-SCTE35:CUE="/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",ID="1074646543",DURATION=255.233', '#EXTINF:6.006,', '1674481974_video_720p-30fps-3436kbps/video_31265.ts']
Vars : {'_lines': ['#EXT-X-KEY:METHOD=AES-128,URI="https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",IV=0xBDFAEF4BF68E82BB3CC3C576E6C11117', '#EXT-X-ASSET:CAID=0x000000003DF78DF3', '#EXT-X-CUE-OUT:255.233', '#EXT-X-SCTE35:CUE="/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",ID="1074646543",DURATION=255.233', '#EXTINF:6.006,', '1674481974_video_720p-30fps-3436kbps/video_31265.ts'], 'media': 'https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts', 'pts': 13036.208689, 'start': 13036.208689, 'end': 13042.214689, 'duration': 6.006, 'cue': '/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==', 'cue_data': {'info_section': {'table_id': '0xfc', 'section_syntax_indicator': False, 'private': False, 'sap_type': '0x3', 'sap_details': 'No Sap Type', 'section_length': 70, 'protocol_version': 0, 'encrypted_packet': False, 'encryption_algorithm': 0, 'pts_adjustment_ticks': 0, 'pts_adjustment': 0.0, 'cw_index': '0x0', 'tier': '0x0', 'splice_command_length': 5, 'splice_command_type': 6, 'descriptor_loop_length': 48, 'crc': '0xf604d2fb'}, 'command': {'command_length': 5, 'command_type': 6, 'name': 'Time Signal', 'time_specified_flag': True, 'pts_time': 13036.045733, 'pts_time_ticks': 1173244116}, 'descriptors': [{'tag': 2, 'descriptor_length': 46, 'name': 'Segmentation Descriptor', 'identifier': 'CUEI', 'components': [], 'segmentation_event_id': '0x400dce0f', 'segmentation_event_cancel_indicator': False, 'program_segmentation_flag': True, 'segmentation_duration_flag': True, 'delivery_not_restricted_flag': False, 'web_delivery_allowed_flag': False, 'no_regional_blackout_flag': False, 'archive_allowed_flag': False, 'device_restrictions': 'Restrict Group 0', 'segmentation_duration': 255.233556, 'segmentation_duration_ticks': 22971020, 'segmentation_message': 'Provider Placement Opportunity Start', 'segmentation_upid_type': 13, 'segmentation_upid_type_name': 'MID', 'segmentation_upid_length': 24, 'segmentation_upid': [{'upid_type': 10, 'upid_type_name': 'EIDR', 'upid_length': 12, 'segmentation_upid': '10.5239/2435-a45b-0-0-0'}, {'upid_type': 8, 'upid_type_name': 'AiringID', 'upid_length': 8, 'segmentation_upid': '0x3df78df3'}], 'segmentation_type_id': 52, 'segment_num': 0, 'segments_expected': 0, 'sub_segment_num': 0, 'sub_segments_expected': 0}]}, 'tags': {'#EXT-X-KEY': {'IV': '0xBDFAEF4BF68E82BB3CC3C576E6C11117', 'URI': 'https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key', 'METHOD': 'AES-128'}, '#EXT-X-ASSET': {'CAID': '0x000000003DF78DF3'}, '#EXT-X-CUE-OUT': 255.233, '#EXT-X-SCTE35': {'DURATION': 255.233, 'ID': '1074646543', 'CUE': '/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w=='}, '#EXTINF': 6.006}, 'tmp': 'noaes-video_31265.ts', 'base_uri': 'https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/', 'last_iv': '0xBDFAEF4BF68E82BB3CC3C576E6C11117', 'last_key_uri': 'https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key', 'debug': True}
{
"media": "https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts",
"pts": 13036.208689,
"start": 13036.208689,
"end": 13042.214689,
"duration": 6.006,
"cue": "/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",
"cue_data": {
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 70,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0x0",
"tier": "0x0",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 48,
"crc": "0xf604d2fb"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 13036.045733,
"pts_time_ticks": 1173244116
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 46,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x400dce0f",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": true,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": false,
"no_regional_blackout_flag": false,
"archive_allowed_flag": false,
"device_restrictions": "Restrict Group 0",
"segmentation_duration": 255.233556,
"segmentation_duration_ticks": 22971020,
"segmentation_message": "Provider Placement Opportunity Start",
"segmentation_upid_type": 13,
"segmentation_upid_type_name": "MID",
"segmentation_upid_length": 24,
"segmentation_upid": [
{
"upid_type": 10,
"upid_type_name": "EIDR",
"upid_length": 12,
"segmentation_upid": "10.5239/2435-a45b-0-0-0"
},
{
"upid_type": 8,
"upid_type_name": "AiringID",
"upid_length": 8,
"segmentation_upid": "0x3df78df3"
}
],
"segmentation_type_id": 52,
"segment_num": 0,
"segments_expected": 0,
"sub_segment_num": 0,
"sub_segments_expected": 0
}
]
},
"tags": {
"#EXT-X-KEY": {
"IV": "0xBDFAEF4BF68E82BB3CC3C576E6C11117",
"URI": "https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",
"METHOD": "AES-128"
},
"#EXT-X-ASSET": {
"CAID": "0x000000003DF78DF3"
},
"#EXT-X-CUE-OUT": 255.233,
"#EXT-X-SCTE35": {
"DURATION": 255.233,
"ID": "1074646543",
"CUE": "/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w=="
},
"#EXTINF": 6.006
},
"tmp": "noaes-video_31265.ts",
"base_uri": "https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/",
"last_iv": "0xBDFAEF4BF68E82BB3CC3C576E6C11117",
"last_key_uri": "https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",
"debug": true
}
{
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 70,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0x0",
"tier": "0x0",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 48,
"crc": "0xf604d2fb"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 13036.045733,
"pts_time_ticks": 1173244116
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 46,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x400dce0f",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": true,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": false,
"no_regional_blackout_flag": false,
"archive_allowed_flag": false,
"device_restrictions": "Restrict Group 0",
"segmentation_duration": 255.233556,
"segmentation_duration_ticks": 22971020,
"segmentation_message": "Provider Placement Opportunity Start",
"segmentation_upid_type": 13,
"segmentation_upid_type_name": "MID",
"segmentation_upid_length": 24,
"segmentation_upid": [
{
"upid_type": 10,
"upid_type_name": "EIDR",
"upid_length": 12,
"segmentation_upid": "10.5239/2435-a45b-0-0-0"
},
{
"upid_type": 8,
"upid_type_name": "AiringID",
"upid_length": 8,
"segmentation_upid": "0x3df78df3"
}
],
"segmentation_type_id": 52,
"segment_num": 0,
"segments_expected": 0,
"sub_segment_num": 0,
"sub_segments_expected": 0
}
]
}
Media: https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts
Lines Read: ['#EXT-X-KEY:METHOD=AES-128,URI="https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",IV=0xBDFAEF4BF68E82BB3CC3C576E6C11117', '#EXT-X-ASSET:CAID=0x000000003DF78DF3', '#EXT-X-CUE-OUT:255.233', '#EXT-X-SCTE35:CUE="/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",ID="1074646543",DURATION=255.233', '#EXTINF:6.006,', '1674481974_video_720p-30fps-3436kbps/video_31265.ts']
Vars : {'_lines': ['#EXT-X-KEY:METHOD=AES-128,URI="https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",IV=0xBDFAEF4BF68E82BB3CC3C576E6C11117', '#EXT-X-ASSET:CAID=0x000000003DF78DF3', '#EXT-X-CUE-OUT:255.233', '#EXT-X-SCTE35:CUE="/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",ID="1074646543",DURATION=255.233', '#EXTINF:6.006,', '1674481974_video_720p-30fps-3436kbps/video_31265.ts'], 'media': 'https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts', 'pts': 13036.208689, 'start': 13036.208689, 'end': 13042.214689, 'duration': 6.006, 'cue': '/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==', 'cue_data': {'info_section': {'table_id': '0xfc', 'section_syntax_indicator': False, 'private': False, 'sap_type': '0x3', 'sap_details': 'No Sap Type', 'section_length': 70, 'protocol_version': 0, 'encrypted_packet': False, 'encryption_algorithm': 0, 'pts_adjustment_ticks': 0, 'pts_adjustment': 0.0, 'cw_index': '0x0', 'tier': '0x0', 'splice_command_length': 5, 'splice_command_type': 6, 'descriptor_loop_length': 48, 'crc': '0xf604d2fb'}, 'command': {'command_length': 5, 'command_type': 6, 'name': 'Time Signal', 'time_specified_flag': True, 'pts_time': 13036.045733, 'pts_time_ticks': 1173244116}, 'descriptors': [{'tag': 2, 'descriptor_length': 46, 'name': 'Segmentation Descriptor', 'identifier': 'CUEI', 'components': [], 'segmentation_event_id': '0x400dce0f', 'segmentation_event_cancel_indicator': False, 'program_segmentation_flag': True, 'segmentation_duration_flag': True, 'delivery_not_restricted_flag': False, 'web_delivery_allowed_flag': False, 'no_regional_blackout_flag': False, 'archive_allowed_flag': False, 'device_restrictions': 'Restrict Group 0', 'segmentation_duration': 255.233556, 'segmentation_duration_ticks': 22971020, 'segmentation_message': 'Provider Placement Opportunity Start', 'segmentation_upid_type': 13, 'segmentation_upid_type_name': 'MID', 'segmentation_upid_length': 24, 'segmentation_upid': [{'upid_type': 10, 'upid_type_name': 'EIDR', 'upid_length': 12, 'segmentation_upid': '10.5239/2435-a45b-0-0-0'}, {'upid_type': 8, 'upid_type_name': 'AiringID', 'upid_length': 8, 'segmentation_upid': '0x3df78df3'}], 'segmentation_type_id': 52, 'segment_num': 0, 'segments_expected': 0, 'sub_segment_num': 0, 'sub_segments_expected': 0}]}, 'tags': {'#EXT-X-KEY': {'IV': '0xBDFAEF4BF68E82BB3CC3C576E6C11117', 'URI': 'https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key', 'METHOD': 'AES-128'}, '#EXT-X-ASSET': {'CAID': '0x000000003DF78DF3'}, '#EXT-X-CUE-OUT': 255.233, '#EXT-X-SCTE35': {'DURATION': 255.233, 'ID': '1074646543', 'CUE': '/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w=='}, '#EXTINF': 6.006}, 'tmp': 'noaes-video_31265.ts', 'base_uri': 'https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/', 'last_iv': '0xBDFAEF4BF68E82BB3CC3C576E6C11117', 'last_key_uri': 'https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key', 'debug': True}
{
"media": "https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/1674481974_video_720p-30fps-3436kbps/video_31265.ts",
"pts": 13036.208689,
"start": 13036.208689,
"end": 13042.214689,
"duration": 6.006,
"cue": "/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w==",
"cue_data": {
"info_section": {
"table_id": "0xfc",
"section_syntax_indicator": false,
"private": false,
"sap_type": "0x3",
"sap_details": "No Sap Type",
"section_length": 70,
"protocol_version": 0,
"encrypted_packet": false,
"encryption_algorithm": 0,
"pts_adjustment_ticks": 0,
"pts_adjustment": 0.0,
"cw_index": "0x0",
"tier": "0x0",
"splice_command_length": 5,
"splice_command_type": 6,
"descriptor_loop_length": 48,
"crc": "0xf604d2fb"
},
"command": {
"command_length": 5,
"command_type": 6,
"name": "Time Signal",
"time_specified_flag": true,
"pts_time": 13036.045733,
"pts_time_ticks": 1173244116
},
"descriptors": [
{
"tag": 2,
"descriptor_length": 46,
"name": "Segmentation Descriptor",
"identifier": "CUEI",
"components": [],
"segmentation_event_id": "0x400dce0f",
"segmentation_event_cancel_indicator": false,
"program_segmentation_flag": true,
"segmentation_duration_flag": true,
"delivery_not_restricted_flag": false,
"web_delivery_allowed_flag": false,
"no_regional_blackout_flag": false,
"archive_allowed_flag": false,
"device_restrictions": "Restrict Group 0",
"segmentation_duration": 255.233556,
"segmentation_duration_ticks": 22971020,
"segmentation_message": "Provider Placement Opportunity Start",
"segmentation_upid_type": 13,
"segmentation_upid_type_name": "MID",
"segmentation_upid_length": 24,
"segmentation_upid": [
{
"upid_type": 10,
"upid_type_name": "EIDR",
"upid_length": 12,
"segmentation_upid": "10.5239/2435-a45b-0-0-0"
},
{
"upid_type": 8,
"upid_type_name": "AiringID",
"upid_length": 8,
"segmentation_upid": "0x3df78df3"
}
],
"segmentation_type_id": 52,
"segment_num": 0,
"segments_expected": 0,
"sub_segment_num": 0,
"sub_segments_expected": 0
}
]
},
"tags": {
"#EXT-X-KEY": {
"IV": "0xBDFAEF4BF68E82BB3CC3C576E6C11117",
"URI": "https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",
"METHOD": "AES-128"
},
"#EXT-X-ASSET": {
"CAID": "0x000000003DF78DF3"
},
"#EXT-X-CUE-OUT": 255.233,
"#EXT-X-SCTE35": {
"DURATION": 255.233,
"ID": "1074646543",
"CUE": "/DBGAAAAAAAAAAAABQaARe5I1AAwAi5DVUVJQA3ODwDAAAFegowNGAoMFHckNaRbAAAAAAAACAgAAAAAPfeN8zQAAAAA9gTS+w=="
},
"#EXTINF": 6.006
},
"tmp": "noaes-video_31265.ts",
"base_uri": "https://turnerlive.warnermediacdn.com/hls/live/586495/cnngo/cnn_slate/",
"last_iv": "0xBDFAEF4BF68E82BB3CC3C576E6C11117",
"last_key_uri": "https://turnerlive.akamaized.net/hls/live/586495/cnngo/cnn_slate/keys/6499af5edf2439cde2a36cfa0086ea80501d4970.key",
"debug": true
}Here's the truth. I was sitting trying to debug a m3u8 in live mode, and I'm thinking, " I need more data so I can figure this out... shit, dude was right" :)
futzu
commented
1 year ago I'm going to close this then.
Sorry man, I didn't mean to piss you off.
I have been failing to decode some #EXT-OATCLS-SCTE35 messages with m3ufu.
My assumption is that I am doing something incorrect, but I would welcome clarification on expected behavior.
Because I do not trust myself, I'll use a SCTE message from Adrian's examples/timesignal/time_signal_placement_opportunity_end.py. I appreciate that this is not a CUE-OUT message, I'm just using a tried and trusted cue message from the threefive samples as an exemplum.
Here's the cue from the threefive repo...
Lets run this cue through
threefive. It decodes the command nicely (and of course it should, because it is one of Adrian's examples)...Lets put it in a bare-bones m3u8 wrapper and put that message in an
#EXT-OATCLS-SCTE35tag...Now lets run m3ufu on that guy...
I was expecting to see:
"cue_data":{}to be populated. It is not."cue":payload to match that of the#EXT-OATCLS-SCTE35. There is an equals sign missing from the B64 payload in the output of m3ufu.But, lets manually put an extra equals character on the end in the HLS manifest, on the assumption that is is getting dropped in the parsing of m3ufu and prior to decode by threefive...
Boom! m3ufu fully decodes the cue this time... One equals character is still dropped, but this time:
"cue_data":{}is populated and reflects the output of threefive"command":{}data now includes what I expected to see."cue":still does not match that of#EXT-OATCLS-SCTE35Question 1: _Is m3ufu intentionally dropping the final equals sign when the tag type is
#EXT-OATCLS-SCTE35? It looks like m3ufu is dropping the final '=' because it is also a delimiter.Question 2, I'm seeing a similar issue with
#EXT-X-CUE-OUT-CONTnot getting decoded intocue_data, both because of the equals issue above and becauseself._do_cue()is not called. Here's a test case that I knocked up.Here's the output, with TEST2,3,4 showing that is the payload contains equals, then BadStuffHappens.
And to prove this, I modified the
#EXT-X-CUE-OUT-CONThandler at https://github.com/futzu/m3ufu/blob/faea00b7bda428bca1b60e37ae8a5c55baa2109a/m3ufu.py#L331 to:self.cue. (I am a bad person and I will go to hell).self._do_cue()which was missing from the#EXT-X-CUE-OUT-CONTcode.With the above crime-against-python,
#EXT-X-CUE-OUT-CONTsuccessfully decoded. It is a dirty proof of concept, but I did manage to get it to producecue_datafor CONT messages. Please excuse me - as you have been painfully aware from our various back and forth, I am not a developer... And I am illiterate when it comes to python. But I hope the above at least demonstrates the issue.So I suspect that there's a wider issue with cues that contain equals character, due to python treating any equal character as being a delimiter during tag parsing. The trouble is that a lot of base-64 SCTE-35 cues contain the equals character.
For ref, here's my current versions...
As ever, BigRespect for your work and the suite of tools.