search

tvkitchen / appliances

A one stop shop for official TV Kitchen Appliances
GNU Lesser General Public License v3.0
3 stars 0 forks source link

STREAM.CONTAINER positions cap out #111

Closed slifty closed 3 years ago

slifty commented 3 years ago

Bug

Current Behavior

After leaving a caption extraction implementation of TVK on for a few hours the STREAM.CONTAINER positions start to break down (which of course breaks down everything else).

I don't know how far it goes, but here's some output of the positions of emitted containers:

STREAM.CONTAINER :: 23860920
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860920
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860920
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860920
STREAM.CONTAINER :: 23860920
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901
STREAM.CONTAINER :: 23860901

Specifically, these two numbers just repeated for hours.

It's not yet clear what's going on but processing broke down on both nights (I don't know if it broke down with that specific number the first night, as I added the logging for the second).

Note that maxint is 2147483647

Expected Behavior

the positions should continue to monotonically increase (until / unless the PTS in the actual broadcast rolls over)

slifty commented 3 years ago

I ran again this morning while out and about and it stopped at:

STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860907
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860928
STREAM.CONTAINER :: 23860907
STREAM.CONTAINER :: 23860907

Not the exact same number but close enough that there's probably a clue somewhere!

slifty commented 3 years ago

So I went into the demuxer and removed the rule that will only update the pst if it is increasing and kaboom:

STREAM.CONTAINER :: 23860884
STREAM.CONTAINER :: 23860860
STREAM.CONTAINER :: 23860860
STREAM.CONTAINER :: 23860884
STREAM.CONTAINER :: 23860884
STREAM.CONTAINER :: -23860908
STREAM.CONTAINER :: -23860908
STREAM.CONTAINER :: -23860908
STREAM.CONTAINER :: -23860908
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875
STREAM.CONTAINER :: -23860875

Classic.

slifty commented 3 years ago

Since I dug around and found it I'll post this here: http://ecee.colorado.edu/~ecen5653/ecen5653/papers/iso13818-1.pdf

It's the MPEGTS spec, page 32 talks about PTS_DTS parsing.

Ultimately the issue is that the demuxer we're using was ported from C, but JavaScript doesn't store numbers as uint_64 which means I think we're both losing precision and apparently hitting an overflow.

This will require a patch to the demuxer, but the quesiton is whether the demuxer needs to be overhauled to use something like BigInt.

slifty commented 3 years ago

One thing unclear right now: Is this happening at...

1) the demux point 2) avro / kafka encoding and decoding 3) somewhere else

I'm going to try generating a payload with a high value and encoding it to test the second

I'm going to ad some debug logging to the demuxer to test the first (but unfortunately I think I'll need wait six hours to get results...)

slifty commented 3 years ago

Quick reminder that the parsed PTS is actually transformed a bit before turned into a position which is what the above output is rooted in. PTS is divided by divided by 90, basically.

Which is just to say that 23860884 translates to 2147479560... which looks pretty familiar! (remember MAXINT is 2147483647)

To be more explicit: the demuxer is parsing numbers that ultimately hit max int. I'm going to open an issue now with the demuxer suggesting they move to BigInt.

slifty commented 3 years ago

The upstream issue for reference: https://github.com/gliese1337/HLS.js/issues/15

slifty commented 3 years ago

I'm going to make these fixes locally (and push them upstream as well once we know they work).

Here's the location of the fork: https://github.com/tvkitchen/utilities/pull/5