White image gets alpha blended as grey image

[CryZe]

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

28.0.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

doesn't apply

OBS Studio Crash Log URL

No response

Expected Behavior

In https://github.com/obsproject/obs-studio/pull/6257 blending modes have been added. And it seems to indeed "mostly" (it does something, not the right thing) fix my issue with LiveSplit not properly being captured transparently. However I feel like there's still "something wrong". If you look at this image:

you'll notice that it's a white square that only contains white pixels. However the opaqueness changes from left being fully opaque to right being transparent. As far as I understand sRGB really is just a curve applied to the colors such that darker colors get more of the 8-bit available per channel as human vision is not actually linear and we can see changes in brightness in dark colors really well. So in the end sRGB often is approximated as pow(color, 2.2). However if we think about it, the RGB values in my image are fully white and thus it's pow(1.0, 2.2), i.e. RGB stays at (1, 1, 1). The alpha however is somewhere between 0 and 1 as it's a gradient, so with 0.5 or so you get pow(0.5, 2.2) = 0.21... but more importantly, it basically just moves the alpha somewhere else between 0 and 1. So in the end what we should see is an image that no matter whether you consider it to be sRGB or not, should look white, but the gradient may look a little different as things might be a little more or less opaque.

Current Behavior

But that's not at all what the blending method in OBS seems to do. Instead I'm getting a grey image:

Steps to Reproduce

1. Add an image source.
2. Select the linked image.
3. Make sure "Apply alpha in linear space is deactivated"
4. Put it above a white background.

Anything else we should know?

This is a clean reproducible continuation of: https://github.com/obsproject/obs-studio/issues/5674

Also: People in Discord are suggesting that SRGB and sRGB are two separate concepts and that this might be related. However no one was able to link a source nor was I able to find any.

This of course as in the previous issue is just a small part of a bigger blending problem in OBS that affects (more or less) all the sources.

[CryZe]

It seems increasingly clear to me that while I have a reasonably good understanding of what the bug is, the whole problem space apparently is really confusing and that even if I hypothetically would create a "perfect Pull Request fixing all of OBS's issues" that it likely can't be merged. So there probably is no way moving forward with this issue until there's a clear understanding (on both sides) of how blending in OBS currently works and how it should work. Therefore I propose to @jpark37 that the best solution forward is to have some session where we talk all of this through (in Discord probably). And like I said, I would even be willing to do all of the work.

[Warchamp7]

Testing on 28.0.3 I get the following results with the supplied test image on a full white color source. Let's also be mindful not to interchange Blending Mode and Blending Method which are two different settings to avoid additional confusion.

Default Settings

Image Source Properties -> Apply alpha in linear space

Source Settings -> Blending Method -> SRGB Off

Apply alpha in linear space + Blending Method SRGB Off

[CryZe]

Right, so there's multiple layers of issues and confusion here:

Apparently the SRGB On / Off setting is more of a workaround as it decides whether the source should be blended as sRGB or not, which honestly is not something the user should ever decide (the sources should just output in either RGB or sRGB and OBS should just blend it as the source outputs it, or force all sources to be sRGB in the first place). You can super easily choose the wrong option there (and often times it is the wrong one by default) and at any given time there is only ever one correct option and the sources know which one it is.

The other setting, which seems to be specific to the image source, when off, essentially does a sRGB transform around the premultiply, which messes up everything (the premultiply lowers the rgb channels to <1.0 and then the sRGB messes up their channels entirely afterwards, in particular because sRGB is not applied to the alpha channel here, so they completely get out of sync). I can't currently tell if that ever results in anything that's not broken.

TL;DR: I'm pretty sure using float3 here is THE bug that causes the issue in the OP: https://github.com/obsproject/obs-studio/blob/89eeeb9c653adc75e75cdd5be9fd507b7f42c1cf/libobs/graphics/srgb.h?q=gs_premultiply_xyza_srgb_loop#L104-L115

Also what makes more sense would be for the image source to have a clean dedicated Color Format (RGB, sRGB, ...) setting and maybe a premultiplication setting (though images are usually stored with straight alpha).

I've also looked into the Renderdoc output and there's many more problems relating to this, especially in the game capture.

However on a positive note: It looks like internally OBS handles sRGB and premultiplication way more cleanly than two years ago. Huge shoutouts to @jpark37 who seems to have done a lot of work here.

[CryZe]

I've since figured out the EXACT problem and it is mostly what I've already perceived:

Premultiplied RGB and premultiplied sRGB are encoded VERY differently.

For straight alpha RGB / sRGB a half transparent white is encoded as: (0xFF, 0xFF, 0xFF, 0x80) For premultiplied RGB a half transparent white is encoded as: (0x80, 0x80, 0x80, 0x80) For premultiplied sRGB a half transparent white is encoded as (0xBC, 0xBC, 0xBC, 0x80)

You can see that while for straight alpha the encoding ends up the same, for premutliplied the encoding is very different. Essentially only for the RGB case is it true that the whitepoint matches the alpha value. For the sRGB case the sRGB curve only gets applied to the R, G, and B channels and not the alpha channel, so they can reach above the alpha channel.

This essentially means it's ESSENTIAL to know whether we are dealing with RGB or sRGB data when premultiplying as this will have a HUGE influence on the R, G, B channels. The image source correctly provides a setting here: Apply alpha in linear space (although the setting is honestly named very confusingly. It should just be called sRGB / RGB or so).

So the premultiplied space needs to know if it's in sRGB or RGB space. The big problem now comes in after. The source will then be blended as RGB or sRGB by OBS. The user can choose this via the Blending Method setting. And that's where everything goes wrong. Since the source HAD to know for premultiplying whether it deals with sRGB or RGB data, since the result is very different, the user now HAS TO MATCH that as otherwise the encoding essentially corrupts. Fortunately in the case of the image setting the user can totally do that. Though honestly they likely won't, as both settings are not very obvious and separate from each other.

So this essentially means in the 4 screenshots posted by @Warchamp7, only 2 of them are correct. While it may look like there's 3 correct ones, one of them only looks correct by accident. Essentially if you mismatch the encoding, the RGB channels are either going to be too bright, or too dark. So in the case of half transparent white, you are going to deal with (0xBC, 0xBC, 0xBC, 0x80), when interpreted as RGB instead of sRGB is a "white that is whiter than white" as the correct half transparent white should be (0x80, 0x80, 0x80, 0x80) in that case. So out of the 4 images posted, only 2 are correct: The two where both settings align.

Which brings me to the conclusion: The blending method SHOULD NOT be a user facing setting. The source has to prepare the premultiplication in a way where it HAS to know whether the image is RGB or sRGB. The user then HAS TO match that setting on the outside of the source (via the blend method) as otherwise the encoding gets corrupted. This is true for all sources and is even wrong by default for the text source.

[PatTheMav]

While I agree that the setting - as it exists - is a decently-sized footgun, it was the good approach chosen to fix a real user issue: With OBS fixing its blending to be more correct, it broke the visual appearance of many users' setups which were handcrafted to work with OBS' "wrong" blending, but we also weren't able to add a "apply alpha in linear space" property to every source.

I guess what we should do from the project's POV is:

1. Define the absolute correct workflow - OBS should blend linearly internally, so how do sources and assets need to be setup and authored to work correctly with that
2. If a source does not or cannot supply its data correctly, how can it be made to do so?
3. If it can be made to do so, should it ever be changeable by the user (another footgun)?
4. If assets are badly encoded, how can they be made to "behave"?
5. Once again, should this be changeable by the user? If so, should it be something that needs to be actively disabled (i.e. users have to actively disable the correct behaviour in a source's property, signalling that they're doing something that is not great and should rather go fix the original issue)

IMO by default OBS should act as if everything is set up "correct" and then evaluate whether we want to tolerate sources/assets that are just wrong. Assets might be wrong, because they were made to "work" with how OBS blended and obviously it's our bad if they stop working once we finally fixed our mistake, but allowing "broken" assets and sources to work moving forward is not sustainable.

[CryZe]

Does anyone have an example of such a broken asset and can link it? I'm not convinced there even are any broken assets. While yes there are images that are encoded in RGB (linear) / sRGB (non-linear), these are two of the four possible cases seen here: https://github.com/obsproject/obs-studio/issues/7623#issuecomment-1285668209 And I'm fully intending for those two cases to still fully work.

The other two are completely non-sensical to me and I don't believe you even could encode assets in a way where those combinations are valid as no other image editing application (or viewer / browser) has such a bug.

[CryZe]

@jpark37 It's been a while, can you weigh in? This issue is essentially blocked waiting on OBS maintainers to move this forward. I've done everything from my side to prove that this is a bug with no positives to it, affecting many users of OBS, ruining any chances of capturing an application that's semi transparent.

Also to reiterate: This bug is entirely unique to OBS, no other application has this bug, and is unrelated to RGB / sRGB which are both valid encodings. There is almost definitely not a single user that "requires" this math bug to be in there. There also can't be any backwards compatibility concerns with assets that might be encoded in a wrong way, as this has nothing to do with how the images are encoded either. This is just a straight up math bug. This really shouldn't be so controversial as to be open for many years now.

[Lordmau5]

After some chatting in the #development channel in the OBS Discord I've went ahead and did some testing in 2 VTuber model capture applications: VSeeFace and VTube Studio

In VSeeFace the image looks like this:

Whereas capturing it with Game Capture and the Transparency option in OBS looks like this:

---

With VTube Studio it's very similar:

And in OBS again:

---

To replicate it in VSeeFace you can grab it from here for free: https://www.vseeface.icu/ They also supply a base cube model to allow for tracking here: https://www.vseeface.icu/Cube.vrm And here's a small video on how to add the image as a prop in VSeeFace: https://streamable.com/i4v6fk

[CryZe]

We have been told to look for further examples of this replicating. Most of the examples in the original issue still replicate https://github.com/obsproject/obs-studio/issues/5674#issuecomment-997466682

Looking at the ones posted by @Lordmau5:

So the VSeeFace example clearly reproduces the bug as well. It's a little unclear if VTube Studio does so as well. Maybe it's worth looking into if it truly doesn't (it more easily reproduces on a white background as that's where the math goes the most wrong). If it doesn't then we really look should look into why. It is possible that it's premultiplied whereas VSeeFace isn't or vice versa.

Here is a rough proposal of how this bug should be fixed:

1. The user facing blend setting, as shown above, needs to be removed entirely as there only ever is one correct setting. (#7647)
2. Each source needs to properly communicate its encoding to OBS such that the correct blending is used (for many sources there is one exact encoding that is always used by that source).
3. Sources such as the image source and game capture need to have settings for specifying the color space of the underlying image (if it can't be retrieved through more automatic means (maybe the underlying capturing APIs know some of that information))

In particular the game capture may even need both settings for the color space (RGB, sRGB, ...) and whether the captured application is using premultiplied colors or not.

[CryZe]

It is actually not entirely necessary to do the first two steps. The third step is really what is blocking the speedrunning community. The former two are basically UX issues where users of OBS are negatively affected through these bugs (whether they know it or not), but they don't necessarily block proper game capturing.

Considering the first two require quite a large refactoring of how color spaces work in regard to each individual source and the low availability of the OBS maintainers, it likely is easier to focus on the third step first as that can much more uncontroversially be merged.

[Tactsohg]

I had same problem and after doing some research I think I figured out how it works. Image Source output = nonlinear(linear(src.rgb * src.a) + linear(dst.rgb) * (1.0 - src.a)) // weird Image Source + Apply alpha in linear space / Game Capture output = nonlinear(linear(src.rgb) * src.a + linear(dst.rgb) * (1.0 - src.a)) // not bad Image Source + Blending Method SRGB Off output = src.rgb * src.a + dst.rgb * (1.0 - src.a) // what i want Image Source + Apply alpha in linear space + Blending Method SRGB Off / Game Capture + Blending Method SRGB Off output = nonlinear(linear(src.rgb) * src.a) + dst.rgb * (1.0 - src.a) // excuse me?? About the function nonlinear(srgb_linear_to_nonlinear) and linear(srgb_nonlinear_to_linear), it can be found in here. It looks like the Blending Method determines whether the foreground and background colors are blended in linear space, but I guess these options are not friendly to most users. And for Game Capture, the Alpha channel is always premultiplied in linear space, I really hope it can provide an option similar to Image Source.

[CryZe]

@jpark37 It seems like simply fixing the game capture unblocks the speedrunning and the VTuber community. This should be a fairly uncontroversial fix. Are you okay with me sending a PR for that? I would still love to discuss the entire color management issue as a whole, because a lot of the time the color blending uses the wrong math by default, which negatively impacts basically every single user of OBS (and I'm not just talking linear vs. non-linear, I'm talking math that is wrong to the point that it's calculating color values that are out of range (> 255), likely even C++ Undefined Behavior). However, if it's too much with a refactor / time investment, I'm absolutely fine with just writing the fix for just the game capture.

[CryZe]

As of https://github.com/obsproject/obs-studio/pull/9999 the game capture on at least Windows now has a setting to capture pre-multiplied alpha. This together with the (broken) blending method setting now technically allow you to capture transparency correctly on at least Windows (though very unintuitively, almost all of these defaults are wrong). This is not to say this issue is fixed in any real way, but it at least on Windows can be worked around enough now.