Backplate texture affected by pixel filter

[stukowski]

I am trying to render an image with a backplate texture to place some 2d graphics behind the 3d objects. I noticed that the background image gets blurred by the renderer's pixelFilter:

With pixelFilter set to OSP_PIXELFILTER_GAUSS:

With pixelFilter set to OSP_PIXELFILTER_POINT:

The backplate's resolution matches the framebuffer resolution and the texture filter is set to OSP_TEXTURE_FILTER_NEAREST. First, I thought the effect may be coming from the denoiser, but these pictures have been rendered without denoising filter.

Is there a way to leave the background image completely unaffected, but still get antialiasing of the rendered objects?

[johguenther]

What would currently work is to blend the backplate in the application (e.g. while displaying OSPRay's framebuffer), using the alpha-channel of the framebuffer (which contains the "opacity" of the rendered scene). A fix in OSPRay could look like this, essentially always using the PixelFilter POINT for the backplate:

diff --git a/modules/cpu/render/pathtracer/PathTracer.ispc b/modules/cpu/render/pathtracer/PathTracer.ispc
index 98208fe58..d86240944 100644
--- a/modules/cpu/render/pathtracer/PathTracer.ispc
+++ b/modules/cpu/render/pathtracer/PathTracer.ispc
@@ -205,7 +205,7 @@ static ScreenSample PathTracer_renderPixel(PathTracer *uniform self,

     screenSample.ray.t = min(screenSample.ray.t, tMax);

-    pathContext.screen = cameraSample.screen;
+    pathContext.screen = cameraSample.pixel_center;
     ScreenSample sample = PathTraceIntegrator_Li(
         pathContext, pathState, screenSample.ray, screenSample.rayCone, ffh);

Yet, the backplate will still be denoised as well when the denoiser is active, which however should be fine for text. We'll think about changing backplate handling, implementing it as a blend step as part of the postprocessing pipeline. Here the drawback is that colored transparency affecting the background won't work with monochrome alpha blending.

[stukowski]

Thanks a lot for the quick reply. Yes, it could be useful to have at least the option to exclude the background plate from the pixel filtering. That's what users of the map_backplate option want most of the time, I would guess.

I also considered doing the compositing in the application and not leaving it to the OSPRay renderer. However, this approach has its own pitfalls, which I was going to describe in a separate GitHub issue. Hope you don't mind if I hijack this thread now and describe two more related problems.

For one, the RGB components of the background color set on the renderer seem to always creep into the filtered pixels at the edges of an object, even if background alpha=0. The effect becomes quite pronounced when camera depth-of-field is used and the object edges are more blurry than usual:

The left picture rendered with the default backgroundColor=(0,0,0,0) looks good on a dark background, but on a light background like above the objects exhibit unnaturally dark edges. So, my follow-up question would be, how can I render a picture with a transparent background such that it will look good when composited on any background, i.e., without knowing the background color a priori? In other words, I would like to produce a rendering (with alpha channel) that can be saved as a PNG file and then placed onto an arbitrary PowerPoint slide, for example.

Maybe you have a better suggestion, but a possible solution I discovered is to reinterpret the framebuffer content of OSPRay as premultiplied alpha (even though it actually is straight alpha, I think) after rendering the image with backgroundColor=(0,0,0,0). For RGBA values with premultiplied alpha, different blending rules apply when combing the foreground image with the background. Here is how it looks (better) when treating the OSPRay framebuffer contents as premultiplied alpha and doing the compositing in the application:

Another thing I noticed when rendering images with a transparent background (0,0,0,0) is that the alpha channel of the framebuffer contains stochastic noise in semi-transparent object regions. I am using the path tracer backend:

While the noise in the RGB channels is successfully eliminated by the denoising filter, the noise in the alpa channel remains. This leads to a problem when composing the semi-transparent image with a white background in the application. The noise reappears in the visible RGB channels (left) - unlike in the image directly rendered onto a white background by OSPRay (right):

[johguenther]

Very elaborated post, thanks! I'll briefly comment

• a fix for the original issue will come soon (backplate to ignore the pixelFilter)
• yes, for compositing at the app side backgroundColor must be black+transparent and the correct blending mode be used (the documentation does not mention anything about pre-multiplied alpha, we should clarify that)
• commit 5b053439025b2857c60a8da7222045edb2cee9f7 added a parameter to denoise the alpha channel as well
• your last example also demonstrates the limitations of alpha-only blending I mentioned with respect to colored transparency: the brightness is different, because alpha is computed as a mix of the RGB transparency filter weights and thus cannot correctly regard the color of the background

[stukowski]

Thanks for the reply.

The addition of alpha channel denoising will be very useful and should visibly improve the output. I'll try to test the new version next week.

the documentation does not mention anything about pre-multiplied alpha, we should clarify that

Yes, a clarification would be great. By the way, my observation is that, for a smaller fraction of pixels in the framebuffer, rendered with background=(0,0,0,0), the RGB components exceed the alpha component. This is not compatible with the pre-multiplied alpha interpretation. I currently deal with this by downscaling the RGB components if necessary so that they become strictly less than or equal to alpha. But I really don't know if this the right way to do it. I could give you a more concrete example if necessary to show in which areas of an image this occurs.

your last example also demonstrates the limitations of alpha-only blending I mentioned with respect to colored transparency: the brightness is different, because alpha is computed as a mix of the RGB transparency filter weights and thus cannot correctly regard the color of the background

I see. The example picture I've posted above (with compositing done by the application), in which the red semi-transparent spheres appear darker, was done still with the conventional straight-alpha compositing method. When I apply the pre-multiplied alpha scheme, they appear lighter (but still noisy due to the missing alpha channel denoising):

I am generally happy with this kind of result. Physical correctness is not the aim in this case, rather clarity for a scientific illustration.

[johguenther]

the RGB components exceed the alpha component. This is not compatible with the pre-multiplied alpha interpretation

This is because alpha is computed as average of the "RGB opacity". Another view is that RGB is HDR and thus can be larger than 1.0.

[stukowski]

the RGB components exceed the alpha component. This is not compatible with the pre-multiplied alpha interpretation

This is because alpha is computed as average of the "RGB opacity". Another view is that RGB is HDR and thus can be larger than 1.0.

Thanks for the explanation. Do you have a recommendation on how to clamp or normalize such color values for display or compositing? My observation is that a pre-multiplied alpha image in which the RGB components exceed alpha leads to “undefined behavior”, e.g. to unexpected colors of the affected pixels - depending on the software used to display the image.

My current solution is:

R' = R * alpha / max(R,G,B,alpha)
G' = G * alpha / max(R,G,B,alpha)
B' = B * alpha / max(R,G,B,alpha)

[johguenther]

depending on the software used to display the image

If blending works in HDR (exceeding 1.0 is fine) then no adjustments are needed. Otherwise, your approach to renormalize the colors is reasonable (just exclude the alpha==0 case).

[stukowski]

When I apply the pre-multiplied alpha scheme, they appear lighter (but still noisy due to the missing alpha channel denoising)

Here the same image rendered with OSPRay 3.2.0, alpha channel denoising, and highest-quality mode - (looks a lot better, thanks!):