Include presentation times in ingested container stream

slifty commented 4 years ago

Description

This PR adds a critical piece of metadata to data extracted from an ingestion stream and sent to kafka: time.

Specifically, it adds 'ptsTime' to the payload object, which is not yet formally supported by the Payload object in @tvkitchen/base-classes but gives us something to work with.

As you can see in issue #45, The amount of R&D that went into this PR is kind of wild, and it's thanks to the efforts of @reefdog @chriszs @traceypooh and @mhirsch that we managed to figure out the best way to do it.

This PR also adds a more complete set of tests to AbstractIngestionEngine now that the functionality is a bit more defined, and constrains the way that we're integrating with ffmpeg which should have an added bonus of preventing some of the security concerns that was offered by passing a variable to spawn.
This PR updates the AbstractIngestionEngine to use the most recent STREAM.CONTAINER dataType, as described in https://github.com/tvkitchen/base/issues/14

Due Diligence Checklist

[x] Tests
[x] Documentation
[x] Consolidated commits
[x] Relevant labels assigned

Steps to Test

yarn install
yarn test

Deploy Notes

yarn install

Related Issues

Resolves #45 Related to #42

slifty commented 4 years ago

A lot has been changed since the last reviews -- this is ready for a re-review!

traceypooh commented 4 years ago

IF for some reason it'd be useful for me to make a small C-based binary that both copies all packets (passthru) and prints first PTS (or all PTS) to stderr, for a given MPEG-TS stream, as a single lightweight binary "one-shot" to reduce some complexity, potentially, LMK! :)

slifty commented 4 years ago

ALL RIGHT.

I'm sorry that this PR has gotten so large.

I've finally wrangled all the rework from the latest round of comments and I think this is close to final form (might even be final form!)

Please give this a re-review at your leisure. Don't hold back about giving feedback -- I know this has been a few iterations but this is really important to get right.

I'm also going to be making two issues based on past feedback once we're ready to merge:

Moving to minipass.
Looking into improving the TSDemuxer API so it fits a bit more naturally into our pipeline architecture.

slifty commented 4 years ago

I got the green light from @reefdog to merge this! GONNA DO IT!

tvkitchen / countertop