Closed ianshade closed 1 year ago
Nice, this is one of things that I have been wondering about, but the benefit was too much of an unknown for me to be able to motivate.
I'll give this and your other PR a proper read next week.
Should we make RLE optional?
I think it would be good to make it opt out. Perhaps always disabled when running in single-threaded mode, as that 30-40ms of cpu time might be long enough for the connection to drop?
Making it opt-out gives a nice escape hatch in case it does cause connection issues, or for when you know that the RLE will do nothing (eg, source is from a jpeg).
I think this should be specified with an options object, so that we can add some other options later, like specifying the input format or output colourspace. (we currently assume its a RGBA buffer, and converting to 709)
Should we try WASM with SIMD for the color space conversion and RLE to squeeze out some more performance?
If we were to move the RGB->YUV into that too then WASM will probably be beneficial. The problem with WASM is that the pixels will need to be copied from a nodejs buffer into a WASM buffer, and the inverse on the way out, so if not careful it could be more costly.
I prototyped a similar conversion in my streamdeck library, which did halve the time of the buffer operations, but the numbers (for 72x72 pixels) were small enough to make the benefit questionable
Patch coverage: 90.47%
and project coverage change: +0.05%
:tada:
Comparison is base (
3a331ff
) 86.23% compared to head (bb9a1dc
) 86.28%. Report is 1 commits behind head on master.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
I have made this opt-out, and this is now released.
I also started playing around with doing the RGBA->YUVA conversion in a native library in #152, we could look at moving the RLE to that too later on, once/if that gets merged
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) Performance improvement for uploading stills and clips that benefit from RLE
What is the current behavior? (You can also link to an open issue here) Frames are sent uncompressed, which consistently takes about 600ms per 1920x1080 frame
What is the new behavior (if this is a feature change)? Encodes still and clip frames with Run-Length Encoding supported by ATEM switchers, making some frames (those that have lots of solid color areas) upload significantly faster (even down to 100ms). Also increases
MAX_PACKETS_TO_SEND_PER_TICK
to 50 which seems to be the sweet spot.Other information: This currently also includes changes from https://github.com/nrkno/sofie-atem-connection/pull/147
On my 12th Gen Intel notebook CPU encoding takes about 30-40ms for a 1920x1080 frame, no matter how compressible it is. Should we try WASM with SIMD for the color space conversion and RLE to squeeze out some more performance? Should we make RLE optional? - I'm open for suggestions.
The code is ready for review, but some help is needed to fix existing tests that I believe have wrong hashes in them.