HDR Workflow Discussion

[latenitefilms]

@JW144754 writes:

Also a question on a separate topic, what is the proper workflow if I want work in HDR in FCP? Is that supported?

To be honest... I have no idea. I don't really know enough about HDR workflows in Final Cut Pro and Motion to know how things should work. Maybe @markspen has some insight?

If it's not working as expected, you could try and open up the BRAW Toolbox Motion Template and tick "Wide Gamut HDR" Color Processing for the Project?

All of the actual "image processing" is done using the BRAW SDK - so it's entirely possible that it already supports HDR out of the box - I'm just not knowledgeable enough in this area to give a definite answer, sorry!

Related:

• Managing Color Space and Gamut in Plug-ins
• Use wide-gamut HDR color processing in Motion
• Use wide-gamut HDR color processing in Final Cut Pro
• Instruct a template to override the Final Cut Pro color space

[latenitefilms]

@markspen writes on Twitter:

should work fine if, like you suggested, the Motion project is set to HDR and it's used in an FCP HDR library and HDR project.

However, based on the below highlighted text, it sounds like if you use a "Standard" colour processing setting in your Motion Template, but insert it in a Wide Gamut HDR FCPX timeline, it'll "force" the Motion Template to "Wide Gamut HDR" colour processing anyway, so you MIGHT not even need to change it?

[JWuStudio]

Yes, the is consistent with what I have experimented so far.I have also discovered that the color space of the Synchronized clip the Toolbox generated is set to rec709, so just need to make sure that matches the timeline colour space, otherwise it gives weird results.

[latenitefilms]

@JW144754 - Given this, are there any changes or improvements we need to do?

[allexino]

I have been thinking about this as well. Unfortunately, I can't think of a good way to do it at the moment. What I need is an idea like the one shown in the diagram.

YUKI

[latenitefilms]

Do you need to attach NCLC tags - can't you just manually tell FCPX things are HDR?

I've just create a new HDR library as a test.

I set my BRAW clip to:

I changed the synchronised clip setting to HDR:

I created a new project/timeline:

I'm not a colourist, but the footage looks ok in the timeline?

Given this, I THINK it all works as-is today - however as an improvement we could automatically set the synchronised clip to "Wide Gamut HDR - Rec. 2020 PQ" if the Color Space/Gamut is Rec.2020 in the Workflow Extension.

[JWuStudio]

Given this, I THINK it all works as-is today - however as an improvement we could automatically set the synchronised clip to "Wide Gamut HDR - Rec. 2020 PQ" if the Color Space/Gamut is Rec.2020 in the Workflow Extension.

Totally agree.

[latenitefilms]

This will be included in the next beta.

If the BRAW Color Space is set to "Rec.2020", and the Gamma is set to "Rec.2100 ST2084 (PQ)", we'll use the "Rec. 2020 PQ" setting in FCPX.

If the BRAW Color Space is set to "Rec.2020", and the Gamma is set to "Rec.2100 Hybrid Log Gamma", we'll use the "Rec. 2020 HLG" setting in FCPX.

If the BRAW Color Space is set to "Rec.2020", and the Gamma is set to something other than PQ or HLG, we'll use the "Rec. 2020" setting in FCPX.

I'll close this issue for now, but feel free to re-open it if you run into any other issues or have any questions.

[JWuStudio]

you are the best! These braw clips looks amazing in rec2020 in FCP. Wish I had this plugin a year ago, would have saved me so much transcode time and I would have switched to HDR delivery then. Thanks for everything again!

[allexino]

Great job!

I will have to try it out, but unfortunately I expect the results will not be as good as desired.

We have already tried the suggested settings, but with no good results. Take a look at this. You can see the difference processing the same BRAW. (processed by DaVinci Resolve)

https://youtu.be/bRQrywh1GIg?t=786

If you have a MacBookPro, iPhone or iPad with a modern HDR monitor, you should be able to see it in HDR. You should get this result.

Back to the original story

The problem with this result in the first place is that the processing in Motioin treats the BRAWToolbox result as Rec. 709(yes, data was changed colorspace,gamma). It is only "Rec.709 in WG" even if you change the project settings.

thanks

[latenitefilms]

My understanding is that if your library and project is HDR, then any Motion Templates should be processed in HDR too.

We're using the official BRAW SDK - so the actual BRAW image processing is all done by Blackmagic - the same "engine" as DaVinci Resolve, BRAW Studio, EditReady, etc.

If you have a Rec.2020 colour space and a Rec.2100 ST2084 (PQ) gamma in BRAW Toolbox, then these rendered pixels is what gets passed to Final Cut Pro via FxPlug4.

AFAIK, it should work exactly the same as BRAW Studio in Premiere (which also uses the official BRAW SDK):

If it's not working for you, then you can try selecting the "Wide Gamut HDR" option in the Motion Template, as per my first post - however, based on what I've read, this isn't required unless you also have "Override FCP Color Space" ticked as well.

As a test, you could try rendering out ProRes 4444 files from Resolve (or via Color Finale Transcoder), and see if they behave differently to BRAW Toolbox clips?

Again, I'm certainly not an expert in this area - so happy to take any suggestions or ideas anyone has!

[latenitefilms]

Here's a clip I've imported into Final Cut Pro via Color Finale Transcoder:

I've imported the same clip with similar settings via BRAW Toolbox:

I have a "Wide Gamut HDR" library, and a "Wide Gamut HDR - Rec. 2020 HLG" project.

BRAW Toolbox definitely does look way darker in the timeline.

I've tried enabling "Wide Gamut HDR" Color Processing in the Motion Template, but it doesn't have any impact.

I'm not exactly sure what's happening, but I don't think it's treating it as Rec 709, because when I use the Rec 709 Display Color Space in Color Finale Transcoder, it looks way brighter, not darker:

Open to suggestions if anyone has any ideas?

[allexino]

You are doing good work.

I understand that this is complicated.

First of all, you need to understand that Rec.709 and Rec.2020 are both 8bit or 10bit,,,,other "data". In other words, Rec. 2020 data can be viewed in Rec. 709. Tag to prove correct color to avoid this problem. There are various types of tags, such as NCLC and other MXF meta.

The BRAW SDK probably does not provide information on tags. Or there is no way to receive it. Hence "data that is Rec.2020 and HDR" is "viewed/processed in Rec.709".

If this processing is not done properly, the saturation will be reduced in the case of gamut and dimmed in the case of HDR.

You are working well ...

thanks

yuki

[allexino]

As for Premiere, Premiere does not have an adequate color management system. It is a rapidly growing product and is far from "management". So there should be nothing you can do about the "color space". It should only support REC.709 and some formats with NCLC tags. The same situation should be true for RED support.

DaVinci Resolve is the strongest CMS, and FCP is the most common level in this generation.

I have never used Color Finale Transcoder, but I think it converts to movie data. It probably embeds NCLC in the conversion.

thanks

yuki

[latenitefilms]

The BRAW SDK basically just gives us a Metal MTLBuffer, in one of the following formats, depending on what Final Cut Pro asks for:

I've tried forcing to blackmagicRawResourceFormatBGRAF32, and it does seem to have a impact:

Still not right or correct, but somewhat closer.

We essentially just pass this MTLBuffer directly from the BRAW SDK to FxPlug4 - there's no real processing that happens, but there could be some kind of bug/mistake/misunderstanding somewhere in the code.

[latenitefilms]

I have never used Color Finale Transcoder, but I think it converts to movie data. It probably embeds NCLC in the conversion.

This is what I can see in MediaInfo:

[allexino]

Thanks for the info. I still think Finale Transcoder is embedding NCLC.

To see the NCLC, I use MediaInfo as well, or look at the information in the Finder.

If it is wrong, I correct it in the FCP inspector.

I wish I could fix this issue in FCP, but I can't set it up for compound clips.

thanks

yuki

[latenitefilms]

Interestingly, if I apply the BRAW Toolbox effect to the MOV that came out of Color Finale Transcoder (as opposed to a Generator, which is what we normally do), it's also less saturated and darker:

As you can see in the screenshot I have the Color Space override to Rec. 2020 PQ as well - although this doesn't matter, as the NCLC tagging for this clips tells FCPX it's BT.2020/PQ anyway.

However, if I modify the BRAW Toolbox code, and instead of passing across the image from the BRAW SDK, I just simply send back the source pixels (i.e. in this case the ProRes 444 from Color Finale Transcoder) - the image looks correct, so I think the FxPlug4 pipeline itself is working correctly. If there was something wrong with the Metal Shader code for example, you'd see a difference.

I think this also confirms that enabling "Wide Gamut HDR" Color Processing in the Motion Template, doesn't actually make a difference in this case.

So it seems the issue isn't that FxPlug4/FCPX can't handle this - the issue is that the image data I'm feeding it from the BRAW SDK is being "clamped" (for lack of a better term) or incorrectly handled.

Color Finale Transcoder also uses the official BRAW SDK - so there's clearly something I'm misunderstanding.

An MTLBuffer is basically just a piece of memory that is used to store data that can be accessed by a Metal compute or graphics processor. The documentation states:

The Metal framework doesn’t know anything about the contents of a MTLBuffer, just its size. You define the format of the data in the buffer and ensure that your app and your shaders know how to read and write the data. For example, you might create a struct in your shader that defines the data you want to store in the buffer and its memory layout.

I can't see anything obvious in the BRAW SDK documentation or header files regarding Wide Gamut HDR. I just assumed that the image data being provided in the MTLBuffer is the "full range" (for lack of a better term) - which I THINK is the case.

I'll reach out to Apple and Blackmagic and see if they have any ideas suggestions, and keep pondering myself.

AFAIK, I don't think NCLC (Non-constant Luminance Color) tagging is relevant in this case - as we're the ones controlling the image data - we're literally just passing MTLTexture's back and forth.

I think the issue is most likely related to a pixel format being wrong SOMEWHERE in the process.

[allexino]

I think on the possibility that it is not a bug. I think the problem is that the Rec. 2020/HDR exported by the BRAW SDK is not being handled properly.

I have tried the following to test it out. Exported as Rec.2020 PQ in the BRAW Toolbox. Compound clip to 709 as in earlier versions. Made it an edit project and exported in ProresLT as 709.

Imported the export again into FCP and specified "Rec.2020 PQ" in "colorspace override". will then get the desired result.

In other words, the data exported by BRAW SDK seems to be OK.

[allexino]

On a different note, I also looked at rec709. I was curious about the highlight clips. Is this a limitation of FXPlug? I cheked on WG709 but same result.

thanks,

yuki

[markspen]

It’s really impossible to evaluate an HDR image without being connected to a calibrated HDR monitor.

Also, assuming you have your clip in an HDR project in and HDR library, you need to be looking at the RGB Waveform set to Nits to see the luminance range with the full scale up to 10K max.

On Jan 7, 2023, at 7:22 AM, allexino @.***> wrote:

On a different note, I also looked at rec709. I was curious about the highlight clips. Is this a limitation of FXPlug? I cheked on WG709 but same result.

thanks,

yuki https://user-images.githubusercontent.com/16377555/211158023-37ad5222-0166-41fd-8eae-c794cb81e1e4.png — Reply to this email directly, view it on GitHub https://github.com/latenitefilms/BRAWToolbox/issues/21#issuecomment-1374514476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYHSAWVSSSEBH2C6VSXI5BLWRGC45ANCNFSM6AAAAAATSNVJPQ. You are receiving this because you were mentioned.

[latenitefilms]

I wonder how the "Color Space Override" toggle actually works? I assume it's just manipulating the pixels. I wonder if it's possible to recreate whatever maths it does in a Motion Template?

[markspen]

I would stay away from color space override - it’s a simple clamp function. If you want to shift between HDR and SDR, much better to use the HDR Tools effect.

On Jan 7, 2023, at 12:40 PM, Chris Hocking @.***> wrote:

I wonder how the "Color Space Override" toggle actually works? I assume it's just manipulating the pixels. I wonder if it's possible to recreate whatever maths it does in a Motion Template?

— Reply to this email directly, view it on GitHub https://github.com/latenitefilms/BRAWToolbox/issues/21#issuecomment-1374604989, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYHSAWTI4IMJDFRCWYLS763WRHIC3ANCNFSM6AAAAAATSNVJPQ. You are receiving this because you were mentioned.

[latenitefilms]

I'm not trying to go from HDR to SDR in this case.

Here's the problem...

I have a Wide Gamut HDR library:

I have a Wide Gamut HDR - Rec. 2020 PQ project:

I've used Color Finale Transcoder to convert a BRAW file to a ProRes 4444 using Rec. 2020 PQ Color Space:

So far so good, this looks good in Final Cut Pro:

I then apply the BRAW Toolbox effect to the exact same clip, and it looks less saturated and darker - I assume because it's "clamped" for some reason:

As discussed earlier, if I tweak the code so that I just pass the "source" image through our effect (i.e. I'm just taking the unmodified data from the ProRes 444 clip and passing it back) - it looks correct - so I think the image pipeline is working properly (i.e. this is still going through our BRAW Toolbox FxPlug4 effect):

Based on what @allexino has said, I think I need to add some kind of transform to "convert" the grey/dark image that we're currently getting out of BRAW Toolbox into something that better matches what we're seeing in the ProRes 444 clip. For example, SOMETHING like this:

This of course is just ASSUMING that all the information is actually there, it's just been "pushed together" (for lack of a better term).

As @allexino notes though - in the above screenshots it does look like the images are been "clipped" over 100 nits(cd/m²) in this library/project. Again, I don't think this is necessarily a limitation of FxPlug4 - because if I pass the "source" footage through our same pipeline, it works fine - as you can see in the screenshot above.

Long story short - I think what we're essentially missing is a "Override Color Space" option for an individual effect. You wouldn't normally ever need this, as effects are normally only ever manipulating source metadata. In this case we're basically using an effect to behave as if it's a generator.

The reason we're using an effect at all (as opposed to just a generator) is that annoyingly Final Cut Pro doesn't update thumbnails for generators - it seems to render them once, and never update, even if you change the generator parameters.

The question then becomes... does any of this actually matter in the real world? By using Final Cut Pro's own colour tools, it looks like I can manipulate the images to get to something very close to what I was seeing in the ProRes 4444:

As @JW144754 mentioned earlier, based on that last screenshot, it seems you can get really nice HDR pictures out of BRAW Toolbox just using FCPX's own colour tools.

I'd be tempted to just leave everything as-is for now, as it does seem to work. However, another option is that we could try and "code" some kind of "curve" into the FxPlug4 that does something similar to what I'm doing with FCPX effects in the above screenshot. However, what exactly that maths would be - I have zero idea.

Of course... I'm not a colourist, and I'm going on the assumption that what Color Finale Transcoder is outputting is correct - however the Color Trix guys are colour geniuses, so I'm assuming they're doing everything correctly.

Based on all of the above, I'm leaning towards just leaving everything as-is - and basically you need to get the RAW setting sitting nicely between 0-100 nits(cd/m²), and then you can use FCPX's Colour Tools to get above 100 nits(cd/m²).

Does that make sense?

[latenitefilms]

Maybe related:

https://fcp.co/forum/4-final-cut-pro-x-fcpx/27832-chromatic-luma-clamping-with-no-processing-added

[latenitefilms]

Jamie writes:

One thing to consider is that when grading HDR, even in FCP, I would never be decoding BRAW directly to an output display transform like PQ REC2020. That would set me up for the wrong order of operations as we always want to grade underneath the output display transform, not on top of it.

I’m always going to be decoding to BMD Film Gen5 BMD wide gamut (or another “scene referred” log space like ARRI LogC3 ARRI wide gamut, or Davinci Intermediate in RCM, or ACEScct AP1 in ACES) and then grading underneath an output transform to 1000 nit PQ REC2020.

[markspen]

Yes exactly correct. Normally (in SDR or HDR) you’d treat BRAW just like ProRes RAW: you’d choose the appropriate gamma/gamut to debayer into - usually a LOG color space - and then can grade or use a LUT or both as desired

On Jan 7, 2023, at 3:09 PM, Chris Hocking @.***> wrote:

Jamie writes:

One thing to consider is that when grading HDR, even in FCP, I would never be decoding BRAW directly to an output display transform like PQ REC2020. That would set me up for the wrong order of operations as we always want to grade underneath the output display transform, not on top of it.

I’m always going to be decoding to BMD Film Gen5 BMD wide gamut (or another “scene referred” log space like ARRI LogC3 ARRI wide gamut, or Davinci Intermediate in RCM, or ACEScct AP1 in ACES) and then grading underneath an output transform to 1000 nit PQ REC2020.

— Reply to this email directly, view it on GitHub https://github.com/latenitefilms/BRAWToolbox/issues/21#issuecomment-1374647247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYHSAWVICZ3DS4RWBBMDCM3WRHZRNANCNFSM6AAAAAATSNVJPQ. You are receiving this because you were mentioned.

[latenitefilms]

Based on the above, keeping it on BMD Gamut and BMD Film Gamma does actually give nice results as well using FCPX's built-in grading tools:

[markspen]

Yeah it looks like you just have a different gamma being applied for the same gamut and I don’t think it’s a huge deail - I’ve never used Color Finale so can’t really comment on that.

On Jan 7, 2023, at 1:58 PM, Chris Hocking @.***> wrote:

I'm not trying to go from HDR to SDR in this case.

Here's the problem...

I have a Wide Gamut HDR library:

https://user-images.githubusercontent.com/22286696/211170593-d98ea078-7d82-4331-8ea3-6d6f3908fc80.png I have a Wide Gamut HDR - Rec. 2020 PQ project:

https://user-images.githubusercontent.com/22286696/211171192-f62bf34a-0e1d-4c16-ab11-c28f43e8f1a4.png I've used Color Finale Transcoder to convert a BRAW file to a ProRes 4444 using Rec. 2020 PQ Color Space:

https://user-images.githubusercontent.com/22286696/211170675-7844d51c-99e1-4fff-917a-00e292977aad.png So far so good, this looks good in Final Cut Pro:

https://user-images.githubusercontent.com/22286696/211170841-800f03d9-cbf1-4f45-b545-5f801f96bc8d.png I then apply the BRAW Toolbox effect to the exact same clip, and it looks less saturated and darker - I assume because it's "clamped" for some reason:

https://user-images.githubusercontent.com/22286696/211171215-d89c0a26-0779-4e8a-9da9-e141fd1f6001.png As discussed earlier, if I tweak the code so that I just pass the "source" image through our effect (i.e. I'm just taking the unmodified data from the ProRes 444 clip and passing it back) - it looks correct - so I think the image pipeline is working properly (i.e. this is still going through our BRAW Toolbox FxPlug4 effect):

https://user-images.githubusercontent.com/22286696/211170954-af7311c6-0dbe-4efa-97e0-e445815ad144.png Based on what @allexino https://github.com/allexino has said, I think I need to add some kind of transform to "convert" the grey/dark image that we're currently getting out of BRAW Toolbox into something that better matches what we're seeing in the ProRes 444 clip. For example, SOMETHING like this:

https://user-images.githubusercontent.com/22286696/211171143-0e65131f-f8b0-481d-9776-f05076e9a196.png This of course is just ASSUMING that all the information is actually there, it's just been "pushed together" (for lack of a better term).

As @allexino https://github.com/allexino notes though - in the above screenshots it does look like the images are been "clipped" over 100 nits(cd/m²) in this library/project. Again, I don't think this is necessarily a limitation of FxPlug4 - because if I pass the "source" footage through our same pipeline, it works fine - as you can see in the screenshot above.

Long story short - I think what we're essentially missing is a "Override Color Space" option for an individual effect. You wouldn't normally ever need this, as effects are normally only ever manipulating source metadata. In this case we're basically using an effect to behave as if it's a generator.

The reason we're using an effect at all (as opposed to just a generator) is that annoyingly Final Cut Pro doesn't update thumbnails for generators - it seems to render them once, and never update, even if you change the generator parameters.

The question then becomes... does any of this actually matter in the real world? By using Final Cut Pro's own colour tools, it looks like I can manipulate the images to get to something very close to what I was seeing in the ProRes 4444:

https://user-images.githubusercontent.com/22286696/211171499-67ddd4b9-6304-4c54-9c43-01ed14b21906.png As @JW144754 https://github.com/JW144754 mentioned earlier, based on that last screenshot, it seems you can get really nice HDR pictures out of BRAW Toolbox just using FCPX's own colour tools.

I'd be tempted to just leave everything as-is for now, as it does seem to work. However, another option is that we could try and "code" some kind of "curve" into the FxPlug4 that does something similar to what I'm doing with FCPX effects in the above screenshot. However, what exactly that maths would be - I have zero idea.

Of course... I'm not a colourist, and I'm going on the assumption that what Color Finale Transcoder is outputting is correct - however the Color Trix guys are colour geniuses, so I'm assuming they're doing everything correctly.

Based on all of the above, I'm leaning towards just leaving everything as-is - and basically you need to get the RAW setting sitting nicely between 0-100 nits(cd/m²), and then you can use FCPX's Colour Tools to get above 100 nits(cd/m²).

Does that make sense?

— Reply to this email directly, view it on GitHub https://github.com/latenitefilms/BRAWToolbox/issues/21#issuecomment-1374626919, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYHSAWTTHMU5P2W2KDAIR4DWRHRHNANCNFSM6AAAAAATSNVJPQ. You are receiving this because you were mentioned.

[markspen]

Again impossible to evaluate that image since it’s either getting clipped or tone-mapped to the display’s luminance capability but it’s good that there’s no clipping in the waveform.

On Jan 7, 2023, at 3:13 PM, Chris Hocking @.***> wrote:

Based on the above, keeping it on BMD Gamut and BMD Film Gamma does actually give nice results as well using FCPX's built-in grading tools:

https://user-images.githubusercontent.com/22286696/211173622-028b1111-376b-45b5-86f1-8139e9b048da.png — Reply to this email directly, view it on GitHub https://github.com/latenitefilms/BRAWToolbox/issues/21#issuecomment-1374648279, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYHSAWSXNWOQWW4CFYO4DCTWRH2CFANCNFSM6AAAAAATSNVJPQ. You are receiving this because you were mentioned.

[latenitefilms]

Yes exactly correct. Normally (in SDR or HDR) you’d treat BRAW just like ProRes RAW: you’d choose the appropriate gamma/gamut to debayer into - usually a LOG color space - and then can grade or use a LUT or both as desired

Given this, I'm not sure there's anything else we need to do for BRAW Toolbox?

[latenitefilms]

Again impossible to evaluate that image since it’s either getting clipped or tone-mapped to the display’s luminance capability but it’s good that there’s no clipping in the waveform.

FWIW - I'm just looking at in on my MacBook Pro (16-inch, 2021) with the Apple XDR Display (P3-1600 nits) preset, so I can see more information in the highlights with my eyes looking at the screen than I can see in the above screenshots.

[latenitefilms]

Note to self: I'll add an option to the new Settings button to enable Wide Gamut HDR settings for the sync clip's and multicam's, rather than trying to "guess" it based on gamut/gamma.

[allexino]

hi

"Color Space Override" simply changes the "tag" indicating which gamut or gamma to use. The FCP looks at its CMS and decides how to display/process it. So it does not change the data. It only changes "how it looks" and "processing result" in FCP. No clipping.

As I mentioned before, "data" is just "data" whether it is Rec709 or HDR.

It is true that the high brightness component (10,000 nits) of HDR cannot be seen visually. It is about 1,000 nits on an iPhone and 1,600 nits on an XDR monitor. However, it can be seen as "data" in Wave form. And these things mean that about 1,000 nits needs to be supported.

Clipping by effects has been a common sight for a long time. As we all know very well, even in FCP7, some effects had limitations. This was often due to RGB or YUV processing. And in those cases, they were clipped at 100% as well. I thought it might be similar to this.

Thanks for your work, I appreciate it.

[latenitefilms]

Totally understand what you're saying and all makes perfect sense.

What I can't understand is that all we pass across to FCPX is a MTLTexture, which doesn't contain any tags or colour space metadata. If I apply the BRAW Toolbox effect to a ProRes clip, which has the HDR tagging, I still get the clipping. However, if I tweak the code and feedback back the source image, no clipping happens. So there's SOMETHING about the data in the MTLTexture that causes FCPX to treat it differently OR the BRAW SDK always "scales" the data to "safe" levels. If that's the case, then the question is, how do I "undo" the scaling? There must be some kind of maths to undo the "compression".

I'll keep playing, and see if I can solve this. In the meantime though, even with your disclaimers, it doesn't seem like we're loosing any data or quality - so I don't think this is a dealbreaker.

[allexino]

hi,

I was looking at a lot of information, but unfortunately I couldn't find anything good. I am not a developer, but I have been looking at the BRAW SDK and have not found any good information.

Rather I found some inconvenience with the compound clips.

I wish there was some convenient filter that could replace the compound clip. It should correspond to the color space specification and be compatible with the General LUT.

thnks

yuki

[latenitefilms]

I tweaked our Metal Shader to basically double the brightness, and that definitely comes across in FCPX:

...so this confirms that FxPlug4 definitely has no issues with higher than 100 nits(cd/m²).

So either the MTLBuffer that comes from the BRAW SDK is restricting things to in-between 0-100 nits(cd/m²) - or there's something wrong in our code that takes this MTLBuffer and feeds it into a MTLTexture, although as mentioned in this thread, we don't really do any processing.

FYI: I've also been told "the sdk does output as data levels rather than video levels".

[allexino]

hi,

I think you are right about the BRAW SDK. As with the Log workflow, the "weird picture" that is happening now is caused by looking at it as Rec. 709, and it should look correct as Rec. 2020 PQ.

But now we have some problems.

The compound clip that is being generated now is "Rec.709 in Rec.2020 PQ", and this state cannot be properly Rec.2020 PQ. Rec.709 clip" and should be designated as a rec.2020 PQ using colorspace override.

However, the current compound clip method does not allow me to change the colorspace to the desired behavior. Even if you could, it would be a "weird Rec.709 in Rec.2020 PQ HDR".

So I thought, it would be nice if there was a filter that could recognize multiple clips together as one clip and specify the color space.

Or, that there should be a function to add a color space to the effected clips.

thanks

[latenitefilms]

As with the Log workflow, the "weird picture" that is happening now is caused by looking at it as Rec. 709, and it should look correct as Rec. 2020 PQ.

Ok, if that's the case, let me see if I can solve.

[latenitefilms]

@endavid - I've been looking at your VidEngine repo - awesome stuff! It seems like you've done a lot of experiments with Wide Gamut HDR. Any tips you can share on this thread?

[endavid]

This is too complicated for me 😅 I know nothing about those video formats, Final Cut Pro, or BRAW.

The only thing I can say is that if the MTLBuffer is set to 32-bit per channel, you shouldn't have problem with the HDR data being clamped to 8-bit, as you already said.

From the issues mentioned in this thread about being too dark, or too bright, or almost good but slightly off, the gamma processing sounds suspicious to me.

When you use 16-bit or 32-bit textures in Metal, I believe the color is set to linear RGB by default, so no gamma is applied. Your whole pipeline should be aware of this. It's not the same to apply some transformation in linear RGB, than in gamma RGB. Even simple downscaling, since the average of two linear colours is not the same as the average after applying the gamma.

There could also be a flag somewhere to apply the gamma back. Linear RGB is not understood by monitors, so the usual thing is to apply the gamma before displaying it. So the app could be showing something wrong even if the Metal buffer is correct. Save the image with the correct profile and compare with what the app shows, to verify they look correct.

There's also a flag to do the inverse gamma conversion, to read textures with the gamma applied but internally work in linear RGB. I suppose this won't be your case, because that's only for 8-bit sRGB textures. But perhaps double check that. You don't want to apply the inverse gamma to something that it's already in linear RGB color space, which should be the case for most HDR formats, I believe.

Sorry I can't be of much help... I've only worked with display-P3 color space, and simple HDR images (.hdr).

[latenitefilms]

@endavid - Thank you so much for the detailed reply - HUGELY appreciated! You've given me a lot of stuff to consider and experiment with - THANK YOU!

[latenitefilms]

The amazing Jamie Lejeune offers some more thoughts via this video:

https://latenitefilms.digitalpigeon.com/shr/MATOgJCdEe2UqAY2bxtbWQ/lOFiTj3Cdeea8aJBNIPBFg

I've also included some text chat for future reference:

OK, so I ran the file you were using in the screenshots through Resolve. And I exported it decoded two ways — one as BMD Film Gen5 and another as REC2020 ST2084 PQ. Those were my baseline reference. And from those I can confirm that the result of BRAW tools is not being interpreted correctly by FCP anytime the library is anything other than standard REC709.

This kind of tricky problem is the main reason why I don’t do anything in HDR in FCP (and I try to avoid doing any color work in it at all). The control over the color pipeline is so limited. As far as I can tell, FCP isn’t getting (or perhaps isn’t accepting) the metadata it would need to identify and color manage the result of the BRAW tools effect correctly.

What’s different is going to shift depending on the BRAW decode option. For example, when I decoded to BMD Film Gen5 in an HDR library FCP got the transfer function correct but the gamut doesn’t appear to match the interpretation of the ProRes I exported out of Resolve. And I can’t tell if that’s because FCP’s color management is interpreting the ProRes file primaries wrong or because it’s got BRAW tools primaries wrong. I’d need to hook up two external monitors from separate computers side by side to be able to compare the output of both apps with fewer variables.

Tough part for me is that whole I know exactly what Resolve is doing to the image, and I can turn all forms of color management off to control the image pipeline completely manually, in contrast FCP is opaque to me about what the color management it is exactly doing under the hood in an HDR library.

That’s the thing. It appears to me that it not getting treated as unmanaged data. FCP seems to be mapping it based on metadata the user can’t control. I’ve never tried to grade in HDR in FCP before because of that lack of control over pipeline, when I could just grade in Resolve instead. So it’s possible I’m doing something wrong here in FCP or I am missing something obvious. I need to redo my setup to eliminate some variables here and spend more time with it so I can figure out what is actually going on in each step of the pipeline.

BTW — I’m viewing it out BMD decklink to Flanders XM551U set to REC2020 ST2084 1000nit PQ

So I've been playing with it in just a standard 709 library, to eliminate FCP's whole HDR pipeline as a variable. And what I see is that any decode which isn't either log or REC709 does not work correctly. The useful one to examine is decode to a linear transfer function, which if the values were being passed through into floating point should not clip, but it is clipping out of BRAW Tools. Granted, few people are ever going to want/need to decode to linear, but decoding to linear in Resolve doesn't result in clipping. So, something different is happening with the processing of the values. FCP's internal pipeline, even in REC709 libraries is floating point. So, if you're looking for what's causing the clamping, I think somewhere in the chain out of BRAW Tools the data is clamping at 0 to 1 instead of remaining floating point.

So one other discovery, which seems obvious in hindsight, is that the color space tagging of the clips does matter in how FCP interprets them, but the color space override option isn't there for the clips coming out of BRAW Tools. If on the REC2020 PQ ProRes clip, I use the color space override option to set it to REC709, the image then matches the result out of BRAW tools with the decode set to REC2020 PQ — this is in a REC709 library.

In an HDR library, if decoding the BRAW Tools to BMD FIlm Gen5, the transfer function matches the ProRes BMD Gen5 file inside a REC2020 PQ project, but the gamut is wrong.

As far as I can tell, in an HDR Library and REC2020 PQ project, the result of BRAW tools, even though the clip is being decoded to BMD Film Gen5, and no additional color management should be done to it, it is being interpreted as if the colorspace override function were set to REC2020. The tell is that if I set the color space override that way on the BMD Gen5 ProRes clip, it now matches the output of BRAW Tools.

And it is the exact same thing if I set the clip in BRAW tools to decode to REC2020 + ST2084 PQ. It will only match the REC2020 + ST2084 PQ ProRes clip if I adjust the color space override on the ProRes clip to REC2020 — that's SDR REC2020, not HDR 2020. So, what is happening perhaps isn't that anything is being clamped, it's that in an HDR library + project, the result out of BRAW Tools is always being forcibly interpreted in FCP as if it's SDR REC2020, regardless of what it actually is. There needs to be some way to set turn off that "colorspace override" function off for the clips coming out of BRAW tools. If you can do that, I think it will work correctly. Does that make sense?

Take a look at the video. The floating point limitation could turn out to be moot. The root cause of the “clamping” you’re seeing is that in an HDR library, FCP is treating all BRAW Tools clips as if they have had a color space override of SDR Rec2020 applied.

I think the floating point thing may affect edge cases, but the root here, as far as I can see, is that no matter what I tried, in a wide gamut HDR library, FCP is color managing the clips added via BRAW Tools as if they are REC2020 SDR. No option to bypass, nor to override.

Odd thing though is when the BRAW Toolbox generator is on the ProRes clips, switching the colorspace override should have no impact, since it would only be affecting the source, which isn’t visible. Yet it does shift the saturation a bit. Very strange. It’s subtle, but it’s there. Watch the vectorscope while changing it. I thought I was seeing phantoms, until I checked the vectorscope in it’s zoomed mode.

Depends on how it was all being done. If the source was 12bit RGB in wide gamut log, then it took it to down to SDR and then back to HDR, all done with mathematical operations (not LUTs) and the whole pipeline from the source is 32 bit floating point. Then, you can be sure nothing is being lost along the way. But if the operations are not floating point, then I’m not sure. It’s possible data could be clipped out in the process or you might get some other quantizing error. For example, if I did that all with CST nodes in Resolve it would work fine. But if I did it with LUTs, since they aren’t floating point, I’d expect problems. From playing with setting the gamma in BRAW tools to linear, it seems to me that the operations being done aren’t in floating point.

I ask Jamie how you'd correct the image in Resolve, and he suggested:

The Color Space Transform nodes just give options to choose of input and output, with some additional options for luminance and gamut mapping, plus OOTFs. The actual math is all hidden under the hood. So this is starting from a decode in Resolve plus processing that mimics what's happening out of BRAW tools (or at least it delivers the same values on waveform and vectorscope as in FCP). The CST node is there, but it's disabled.

When I enable that CST node, the result is identical to BRAW directly decoded to REC2020 + ST2084 PQ 1000 nit

In Resolve these are all happening in floating point. That unassuming "apply Inverse OOTF" checkbox is the key attribute. Without it, it's all wrong. Oops. One mistake. The “203 nits diffuse white” should have been checked since, AFAIK, FCP is always calculating that way. Resolve sets the OOTF checkbox automatically according to how the gamma options have been set. OOTF is also sometimes called “system gamma”. And since the CST in the screenshot is going from SDR to HDR, that old “system gamma” that was added in the signal chain to the display needs to inverted, hence the “apply inverse OOTF”.

For BRAW Tools clips decoded to BMD Gen5, it would take inverting the incorrect gamut interpretation, rather than the gamma.

If I import the C216_BMDFIlmGen5-002 clip from Resolve, set it to Rec. 2020, then apply my BRAW Toolbox clip to it, then results are pretty much identical:

So essentially, BRAW Toolbox is outputting SDR "Rec. 2020" (when in a HDR library and project).

I also chatted to Marc Bach, who was grading with a broadcast quality HDR monitor - and he was able to use the colour wheels to "eye" match a BRAW Toolbox clip to match a BRAW clip in Resolve using the colour wheels. He couldn't do it just with curves alone - the changes aren't linear, hence the need for colour wheels. However, he was able to get a match - and confirmed we're not actually loosing any information - the image is just being "compressed" down to SDR "Rec. 2020", and we essentially need to "stretch it out" again.

However, I'm still not sure how to solve this problem. I've tried using a MPSImageConversion Metal Performance Shader to change colour spaces of the MTLTexture, but I can't get it to match.

For example, I tried (and yes, I know it looks like I have source and destination around the wrong way):

CGColorSpaceRef srcColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceITUR_2100_PQ);
CGColorSpaceRef dstColorSpace = CGColorSpaceCreateWithName(kCGColorSpaceITUR_2020);

And I get:

It looks like it's clipping above 100 - not sure why.

I can confirm that both the input texture and output texture is MTLPixelFormatRGBA16Float. I'm currently forcing the BRAW SDK to output in 32-bit (Floating point interleaved RGB).

Resources:

• Processing HDR Images with Metal
• Chapter 6: Confused about sRGB color space and gamma correction in Metal

[allexino]

I almost agree with Jamie Lejeune and Marc Bach. That does not change my opinion so far. I am using DaVinci Resolve with RCM and the results are the same as his.

[latenitefilms]

Blackmagic suggest:

Resolve is 32bit float so using 0-1 type ranges too, just the scopes display as 10bit by default. With Resolve there are settings to tell it to treat the clip as HDR that the end user sets (output space), I think somehow you need to inform FCP the data needs to be interpreted as PQ/HLG/etc. Then it can use ColorSync with its viewer to be displayed appropriately for compatible displays like XDR monitors etc

As discussed several times above, I think if this was just a regular clip, I could just use the “Override Color Space” option to make it work - but because I’m doing this all in an effect, there’s no option for this. I think I’ll need to do the equivalent maths in my Metal Shader - essentially rescale the [0,1] ranges to 16-bit or 32-bit ranges.

Resources:

• Converting RGB values in [0,1] range to high dynamic range (EXR format)
• HDR Scattering and Tone Mapping in Unity

[latenitefilms]

This gets me pretty close in 16-bit:

colorSample *= half4(2^16-1);
colorSample.rgb = colorSample.rgb / half3(255);

[latenitefilms]

The amazing Marc Bach offers some more helpful insight:

Ok! Good news!! Check the screenshot and this gives me good results in my monitor.

This means the image needs to be corrected to PQ.

BTW this is the LUT from ARRI’s website

The moment I changed the Output of the LUT plugin to PQ is when the image showed correctly

The issue remains on the color space as is still not correct

this LUT from ARRI mapped from Rec2020 to Rec2020PQ matches what I see in Resolve on color space and gamma.

Scratch that I had the wrong LUT on Resolve

Setting BRAW to ARRI settings and then applying the same LUT gives matching results on both apps

on color space and gamma

So if we had a BRAW LUT for the right setting TO rec2020 PQ, we could apply that as a Custom LUT and have proper representation of the image

More success! I went to Resolve and set BRAW to DWG Intermediate and then added a Color Transform converting from DWG Intermediate to Rec2020 PQ and exported a LUT of that, then went to FCPX matched the settings and added the LUT and we have a MATCH!!!!

Even more success! Followed the same procedure and set the Transform to 2020 PQ to 2020 PQ and then exported a (null) LUT. Then set those settings on FCPX and used the LUT with Input Rec2020 Output Rec2020PQ and I have a match

So loooong story short, FCPX needs to be tricked on the Custom LUT plugin to go from input rec2020 to output Rec2020 PQ and the image is then normalized for HDR.

Now I get best results image wise using ARRI or DWG Intermediate than using Rec2020 PQ as BRAW settings (with the corresponding LUT applied like ARRI, DWG or 2020PQ null)

That means that you Toolbox is being normalized for 100 nits somewhere

There should be something in the FXPlug API to set the plug-in to PQ instead of SDR or if that doesn’t exist I’d use this workaround with DWG Intermediate with its LUT for HDR projects.

Note to self - these Shaders and this repo look really interesting:

https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Shaders/Shaders.metal

Making PNG LUTs:

https://forum.blackmagicdesign.com/viewtopic.php?f=18&t=35099

This looks useful:

https://videovillage.co/lattice/

[latenitefilms]

Using a "Null LUT" (i.e. a LUT that doesn't actually do anything) in the Custom LUT tool, then setting it to a Rec. 2020 input and a Rec. 2020 PQ output definitely seems to do the trick. For example:

ProRes export from Resolve:

BRAW Toolbox:

I'm now attempting to re-create this Rec. 2002 to Rec. 2020 PQ conversion in code.

Using this code is getting closer:

//---------------------------------------------------------
// Convert SDR to HDR using PQ function
//---------------------------------------------------------
half3 hdrColor = colorSample.rgb;
hdrColor = hdrColor.rgb * pow(2.0, (hdrColor - colorSample.rgb)/0.05);

//---------------------------------------------------------
// We return the color of the texture:
//---------------------------------------------------------
return float4(float3(hdrColor), 1.0);

Changing to hdrColor = hdrColor.rgb * pow(2.0, (hdrColor - 0.5)/0.5); gives me this:

Changing to hdrColor = hdrColor.rgb * pow(2.0, (hdrColor - 0)/0.5); gives me:

Changing to hdrColor = hdrColor.rgb * pow(2.0, (hdrColor - 0.1)/0.4); gives me:

It's SO close - but not quite right yet. Will keep playing.

[latenitefilms]

Jamie Lejeune writes:

It’s a good question about what’s really happening inside FCP’s pipeline. As far as I can tell, with a null LUT and the settings with input as REC2020 and output as REC2020 PQ, all it is doing is changing the transfer function from SDR (whether that’s gamma 2.4, gamma 2.2, or the strict inverse of REC709 camera gamma is unclear, perhaps it’s documented somewhere) to 1000nit PQ. So it’s changing how peak brightness is defined from 100 nit to 1000 nit, and also the transfer function curve from whichever SDR gamma is being used to the PQ curve. Hence, it’s not just a linear scaling. Though there is something going on with the gamut transform too on the log decodes, but one problem at at time

[latenitefilms]

The amazing Marc Bach writes:

So after testing all over again here’s my report:

Import the clip in the HDR settings you want to work.

Apply the Custom LUT with a null LUT with Input Rec2020 and Output matching your HDR flavor.

Doing that works. Changing the BRAW settings inside the synced clip after import and jumping outside doesn’t seem to work correctly because it looks like the synced clip container retains the import color space and gamma settings even after changing them in the inspector.

Now for best color results I find setting BRAW to DWG Intermediate, import and then applying Custom LUT and the corresponding LUT with Input Rec2020 and Output matching your HDR flavor is what gives the best picture results. The corresponding LUTs should be DWG Intermediate to PQ or DWG Intermediate to HLG. I created those LUTs with the Color Transform plugin in Resolve, I can share those with you if you need them.

All this is a workaround, the issue still remains in the BRAW Tool seems to be stuck in Gamma 2.4 no matter the setting you choose.

Let me add that when I say for best results use DWG Intermediate I mean on any app, including Resolve. The mapping of brightness (midtones and shadows) matches better to the Rec709 reference or what it was seen on the camera monitor. Raw settings on PQ give mids that are too dark and HLG too bright (as a simple explanation).

The incredible Jamie Lejeune shares two more videos:

https://latenitefilms.digitalpigeon.com/shr/Sb9mQJJREe2ATAb5sRTbbQ/VU8eFxAFhvBFERfD3IUW1Q

It appears that Marc and I have arrived at exactly the same conclusion of what’s going on and how to solve it, though he has gone further by recommending grading strategies while my solution was solely focused on simply matching the BRAW Tools output to the ProRes transcodes.

The reason for the mids thing Marc mentioned is that the current practice is to use 203 nits as the level for diffuse white, but that wasn’t part of the original ST2084 PQ and HLG specs since the assumption at the time was diffuse white would be 100 nits.

Blackmagic writes:

We do have LUTs to PQ, or I can generate some, but I also saw your note in Github that an identity LUT is enough to trick FCP into applying the correct transforms ?

I saw you are also playing with rescaling our PQ output - maybe a more direct path would be if the user selects PQ gamma then in the background you actually set the output gamma to linear (which will be scene referred where 18% grey card maps to 0.18f and diffuse white ~= 1.f with specular's above 1.f), then apply your own PQ curve that's normalised to the range that will work in FCPX?

Also be aware our BRAW SDK PQ curve still uses the older 100nit diffuse white reference (and max 10000nits), rather than the newer 203nit white if you are matching to that.

I've also sent Jamie & Marc a slightly different version of BRAW Toolbox which uses a RGB linear (not gamma corrected) working color space, instead of RGB at video gamma (2.2) working color space for testing. This may be more appropriate, but I originally avoided it, because with the RGB Gamma option, SDR seemed to look correct right out of the box. This is discussed here in the Set the Desired Color Space section:

https://developer.apple.com/documentation/professional_video_applications/fxplug/managing_color_space_and_gamut_in_plug-ins?language=objc

It's also worth noting that that same link has a section on Determine the Color Gamut in Use for the Working Color Space, where you can use this code to determine the working space of the project:

id<FxColorGamutAPI> colorApi = [_apiManager apiForProtocol: @protocol(FxColorGamutAPI)];
BOOL isHDR = ([colorApi colorPrimaries] == kFxColorPrimaries_Rec2020);
if (isHDR)
{
   // ...
}

HOWEVER, I couldn't get it to work - it always returned kFxColorPrimaries_Rec709, regardless of the clip, project and libraries settings (i.e. if I had everything set to HDR PQ - it still returned 709). I contacted Apple about this a few days ago, but never heard back.

Basically, I think at this point, the best solution is to keep the FxPlug4 in the RGB at video gamma (2.2) working color space, and go back to what I suggested here and add some options to the Settings button for:

COLOR SPACE:

• Standard - Rec. 709
• Wide Gamut - Rec. 2020
• Wide Gamut HDR - Rec. 2020 PQ
• Wide Gamut HDR - Rec. 2020 HLG

...then if the user selects PQ or HLG we automagically add the Custom LUT with a null LUT and the Rec. 2002 to Rec. 2002 PQ/HLG conversion. I don't THINK we need to do anything for Rec. 709 or Rec. 2020.

This will also require adding another button to the main BRAW Toolbox app to install the Null LUT.

I do note that Marc says:

All this is a workaround, the issue still remains in the BRAW Tool seems to be stuck in Gamma 2.4 no matter the setting you choose.

However, in a Rec. 709 project, if I have the Gamma set to Linear, it definitely looks Linear, and I can still grade it using the Color Wheels to a nice image:

Another interesting observation - you can only have Rec. 709 projects in a Standard library:

If you want a SDR Rec. 2020 project, it needs to be in a Wide Gamut HDR library.

Here I'm creating a Wide Gamut - Rec. 2020 project in a Wide Gamut HDR library:

You'll get this message if you try and put HDR clips in this project:

I only note this, as it is a bit confusing, because it makes it sound like Rec. 2020 is only for HDR projects, and yet Wide Gamut - Rec. 2020 is still actually just SDR.

[latenitefilms]

These changes will be actioned in the next public beta.

I'm going to close this issue for now. Thanks team!