Support for Tone Mapping

[softworkz]

Hi,

is it planned to add support for tone-mapping through the libva filter VAProcFilterHighDynamicRangeToneMapping?

Thanks, softworkz

[dmitryermilov]

Right now HDR is missing VPP feature. It's a known gap. @nkurina , @FurongZhang , do we plan to enable it?

[FurongZhang]

We had a plan to do that. Given resource limitation, we planned it to Q2'2020. BTW, @softworkz , Do you have some specific requirement about this? Thanks a lot in advance!

[softworkz]

@FurongZhang - Thanks a lot for responding.

We are Emby (https://emby.media/) - our mission is to allow users to play back their own media on any device and over any kind network connection in the best possible way.

Precise adaption like that requires real-time transcoding in many cases. While we do support quite a range of video hardware acceleration technologies for decoding, processing and encoding, one of our primary targets is Intel QuickSync on both Linux and Windows. We are consuming QuickSync features via a customized ffmpeg, and once there was a tonemapping VPP available in the Intel Media SDK, we could add the parts on the ffmpeg side on our own.

For the reasoning behind: During the past two years, video content encoded as 10bit HDR has become increasingly popular, even though many devices are not actually capable to decode or properly display content in that color space. Hard conversion of color range, often leads to colors appearing as "washed out" or visible banding effects. So, the immediate goal to achieve is to be able to convert 10bit HDR video content to 8bit SDR allowing it to be watched on non-HDR client devices in an acceptable way.

The mentioned kind of video content often comes at 4k resolutions. For many devices not having the most powerful kinds of Intel CPUs, the required tonemapping processing cannot be handled by the CPU (in realtime). That's why there's an urgent need for being able to do this as part of the hardware video processing pipeline.

QuickSync (Intel Media SDK) has always been presented as the "bigger" and more capable "brother" of plain VAAPI. That's why I was surprised to see that VAAPI is already capable of doing tonemapping (there already exists an ffmpeg filter for it) while there's nothing available for it at the side of Intel Media SDK.

Please let me know whether you got any further questions..

[dmitryermilov]

Thank you @softworkz for detailed problem statement. And thank you for accolade towards MediaSDK. Yes, we do understand importance of HDR.

[softworkz]

Hi Dmitry,

you closed the issue, does it mean that it's implemented already?

Best regards, softworkz

[dmitryermilov]

Hi @softworkz

I agree with you. Let tag "feature request" and keep it open.

[softworkz]

Great, thanks!

[FurongZhang]

We are working on that now!

[softworkz]

That's awesome news! Let us know when you got something..

[softworkz]

@FurongZhang - Any news on that subject?

[FurongZhang]

Still WIP on that. Target to release in Q1 in the best case, Q2 in the worst case.

[softworkz]

Thanks for the update!

One question: Will it be a common VPP component or will it be limited to the most recent CPU/GPU models only?

[FurongZhang]

May I know what platform you are requesting?

[softworkz]

Both, Windows and Linux, but most importantly Windows (where there's no VAAPI)..

[FurongZhang]

I mean GPU. Which generation of GPU you are targeting? And BTW, I have defined HDR10 VAAPI, it is in the master branch. Please check the latest VAAPI vpp.h if needed.

[softworkz]

It would be so cool and easy if we had a certain generation to target...

But in fact, our software is made to be installed on many platforms with all different kinds of hardware. We are using ffmpeg and on top of that, we're having a custom detection layer that precisely tells us what capabilities are available on each platform.

I had recently posted a few details for some of your Intel fellows as part of another discussion (= adding DX11 support for QuickSync on Windows) here: https://github.com/softworkz/ffmpeg_dx11/issues This should explain our use cases a bit better.

As soon as you got tone mapping support added as a MediaSDK feature, we will go and add an appropriate VPP filter to ffmpeg (unless somebody else would be faster ;-)

The reason why I'm asking for the hardware scope of that feature is for knowing whether it will cover a significant range of existing hardware or whether it will be just be targeting the cutting-edge of latest hardware that nobody (and none of our customers) has available yet?

[uartie]

https://github.com/intel/media-driver#video-processing-features shows supported platforms for tone mapping on linux driver. Oddly, it's not listed as a feature on KBL... but, I enabled tone map in gstreamer's vaapipostproc element and it works on that platform for me.

[softworkz]

@uartie - thanks for the reference, I know that feature matrix, though.

In the past, Intel Media SDK has usually been providing a superset of hardware video acceleration features compared to plain VAAPI. The process of tone-mapping requires a lot of calculations to be performed, but those calculations are not really special in a way that it would take a specific kind of hardware to process.

And from there, knowing that it doesn't really depend on any underlying hardware capability, I was wondering (+ hoping, at least considering worth asking) whether the QuickSync/libmfx/VPP implementation might have different requirements than VAAPI.

And in case of Windows, there's no VAAPI driver under the hood anyway, so it might differ in requirements as well.

Or not..

That's what I'm asking ;-)

[softworkz]

That's what I'm asking ;-)

In case, it wasn't clear enough: how will the device support for this feature`be on Windows?