In libyuv RGB24 actually means bgr byte order. Hyperion/HyperHDR expect pixels to be in rgb byte order.
This wasn't really an issue with UI layer captures as:
Capture using FMT_ABGR (which is in fact ARGB) => ABGRToARGB (now the bytes are indeed bgr) => ARGBToRGB24 (reversing it again to get rgb again)
Though in case for the captured content from vtcapture we assumed it to be NV21 (which is in fact NV12, confirmed using kernel sources) it kind of works because NV21 uses a similar but swapped layout. However we end up with swapped UV planes resulting in the colors being a little bit off. If NV12 is used then colors are almost perfect.
In libyuv RGB24 actually means bgr byte order. Hyperion/HyperHDR expect pixels to be in rgb byte order.
This wasn't really an issue with UI layer captures as:
Capture using
FMT_ABGR(which is in fact ARGB) => ABGRToARGB (now the bytes are indeed bgr) => ARGBToRGB24 (reversing it again to get rgb again)Though in case for the captured content from vtcapture we assumed it to be NV21 (which is in fact NV12, confirmed using kernel sources) it kind of works because NV21 uses a similar but swapped layout. However we end up with swapped UV planes resulting in the colors being a little bit off. If NV12 is used then colors are almost perfect.
Thanks @asturel for investigating this