Handle Associated alpha images

[Quackdoc]

Describe the bug The main viewer fails to render an image with "zero alpha" pixels. However the zoom preview properly renders

To Reproduce Steps to reproduce the behavior:

1. Load image (provided)
2. Move cursor near candle

Expected behavior you should see a glow around the image

Screenshots / sample files https://files.catbox.moe/477nre.jxl

Desktop (please complete the following information):

• OS: Arch
• [Linux] Cosmic (wayland)
• Version https://github.com/woelper/oculante/pull/314

Additional Context Images encoded from https://twitter.com/jkierbel/status/1210235542404775945

[woelper]

Just to clarify, this is not in master, right?

[Quackdoc]

Just tested master, occuring there too

[Quackdoc]

it's probably worth noting that when viewing RGB the pixels do show up, but since they don't have the right transparency it's blown right out, I think this is why the preview screen "seems" to work

[woelper]

Hmm, I tested the image on master and it displayed fine (meaning the transparent pixels were invisible). Could you send a screenshot please? Maybe this is OS specific?

[woelper]

I made some comparisons, and lots of programs seem to have different ways of handling the alpha:

[woelper]

Oh, I see. I looked at the original twitter thread.

If this is in regular 8 bit per channel, you can see that there is no alpha on the candles - that means they won't display. I've inspected them and they are 0 - so what Oculante does seems correct.

In the repo non-occlusive luminescent pixels are mentioned. I find this a bit esoteric - in the end this is all image data. I suspect the point of this is to say "if an image value is very bright, it should display even if there is no alpha"?

It also depends on how you do the conversion from EXR to JXL. Pixels with no alpha might be handled differently and the resulting JXL should have a higher bit depth than 8bit for this to work if I understand this correctly.

[Quackdoc]

the bitdepth is not an issue, these are 0 alpha pixels.

Without getting to technical, Associated alpha is fundamentally different from non associated alpha. Associated alpha sort of "emulates" how light or rather energy actually works. "Invisible" pixels still have color, and can impart color onto it's background. "Straight" alpha however is a simple modifier on "how much the foreground effects the background".

I'm not sure how easy it is to contextualize, but what I do is in the case of "Straight alpha" you can think of (1,1,1, 1) the "color of the pixel and how much of it I can see". Where as with associated alpha (1,1,1,1) is "how much energy the pixel has, and how much it blocks the view of whats behind it"

I suspect the point of this is to say "if an image value is very bright, it should display even if there is no alpha"?

not quite, it means that "the pixel imparts the energy on what's behind it". (see the below image for an example)

the math for it is a bit different. "Straight alpha" that PNG's use is something like r = fg * fg.alpha + bg * (1 - fg.alpha)

associated alpha on the other hand is something like r = fg + bg * (1 - fg.alpha)

(my math might be a bit wrong it's been a long time)

You can see here that with associated alpha, like the name implies, is actually associated with the color of the final result. 0 alpha doesn't mean the foreground is just invisible (Note associated alpha is also and somewhat more commonly called "premultiplied alpha" which is a bit of a misnomer

you can read more upon it here with this collection of resources https://twitter.com/troy_s/status/1102283028896731136

Image to help give an example

In this case the square is a single color square behind the candle, and the candle impart's it's energy unto the square giving it a tint which in this case is more or less a gradiant. The star is a composited on top of the flame but with a low opacity and a blur applies to the star itself, the flame is extremely bright so it doesn't impart any energy onto the flame since, on the contrary, the flame shines through the star, and while the energy radiated off the flame is significantly less, where it is higher intensity you can see it through the star a bit, but where it's less intense you cannot.

Hopefully this can showcase that "Associated alpha" Is not just "the color of a pixel" but rather more accurately represents how color actually works.

[woelper]

I am glad for the discussion - working in the video game industry I have had my own share of (un)premultiplied image discussions as well - so it's fine to get technical.

I think in the end, the result of your image is achieved by interpreting image data and blending it with something.

My issue is that I don't want this image viewer to be too opinionated. Sometimes you might want to encode something in the alpha channel that is not really alpha - for example you just want to use all 4 channels of an image to store information for a shader. Others might want to extend the color channel by doing alpha bleeding like in this example: which would probably look very strange when being interpreted as associated alpha - at least if you use r = fg + bg * (1 - fg.alpha) which would amount to 1 + 0 * 1 = 1 for a bleeding green pixel over zero alpha.

What we could do is to offer an option to interpret the source in a certain way - for example "treat source as associated alpha" and then we would use a different blend mode on the texture. But for that to work, your image would really need to be premultiplied (basically black where there is no light + alpha). You can get a similar functionality if you select "RGB" as channels in Oculante.

the bitdepth is not an issue, these are 0 alpha pixels.

I am afraid that might not be the case. I've opened the original EXR, which is 16 bit, and it blends correctly in Krita:

As soon as you convert that to 8 bit, this is the result:

I suspect that is because at the candle flame, alpha is 0, but the color value is greater than 1 - so the blend mode in Krita still transfers value to the next layer according to how much brighter it is. We'd have to look at the code of the "Normal" Krita layer blend mode to be sure though - but my guess is that if alpha = 0.0 and the candle is 1.5 you would get 0.5 blended to the layer below.

[Quackdoc]

The image also will render correctly in managed tooling like the screenshot of olive above. Krita has no proper color management (It can't even blend correctly) and GIMP's color management is poor at best (the image doesn't even get loaded as a linear image)

Krita can't handle associated alpha as per the popup when you load the EXR "The image contains pixels with zero alpha channel and non-zero color channels. Krita has modified those pixels to have at least some alpha..."

GIMP also fails to load the EXR properly as when you load it as is, and throw a layer under it, it too doesn't blend correctly.

If you load the jxl image in a program like darktable that has somewhat proper color management, you can see that the image does render as the olive does

EXR is special because it allows values greater then 1, but it's not a necessity for this.

What we could do is to offer an option to interpret the source in a certain way - for example "treat source as associated alpha" and then we would use a different blend mode on the texture.

Indeed, Associated alpha should be "toggle-able" and should be default for a few image formats

EXR is always associated alpha as per the spec, TIFF and JXL tag whether an image uses associated alpha or not, PNG is never associated alpha, PPM based formats do nothing to notify whether they are associated or not, so it's best to treat them as straight alpha, but give a toggle to optionally treat it as associated alpha,

You can get a similar functionality if you select "RGB" as channels in Oculante.

This may actually work, though all my test images are in linear so i'm not sure how well this actually works out since oculante renders everything in srgb still

[woelper]

This may actually work, though all my test images are in linear so i'm not sure how well this actually works out since oculante renders everything in srgb still

I think this is a separate issue - if you like, please open a new ticket and we can map out how we do that properly!

Indeed, Associated alpha should be "toggle-able" and should be default for a few image formats

I agree on the toggle. I'd have to see if I have access to the metadata in TIFF and JXL to check for their alpha handling. We could go as far and even store association, for example "open Png as associated" if that fits your workflow best.

[Quackdoc]

I think this is a separate issue - if you like, please open a new ticket and we can map out how we do that properly!

this would be part of the colormanagement issue

I agree on the toggle. I'd have to see if I have access to the metadata in TIFF and JXL to check for their alpha handling. We could go as far and even store association, for example "open Png as associated" if that fits your workflow best.

you already have a section for alpha tools, toggling associated alpha could be done right from that menu.

as for tiff and jxl

jxl-oxide uses Alpha::associated_alpha from jxl_oxide::ExtraChannelType

tiff uses, and don't shoot the messeneger, I have no idea what they named it this way the extrasamples from the enum here tiff::tags::Tag

see page 31 https://www.itu.int/itudoc/itu-t/com16/tiff-fx/docs/tiff6.pdf this is the actual listing in the spec. strange name but ok.