Transparent OOB Values

[brodieG]

It would be awesome to include an alpha channel in color values (or something like it) such that if a ray comes from e.g. infinity it gets recorded as an alpha value of 0. That way antialiasing will work even when a rendered image is posted on different color backgrounds (e.g. right now I can't make my website darkmode without all the rayrendered images sticking out like sore thumbs because of the white background I blend them into). This might also work with transparent materials so that you can "see" the background through them.

Background here is e.g. the color of the HTML page the rendered images are posted on.

I might be the only person who wants this as one of the few that prefers rendering objects stand alone, but wanted to plant the seed in case this appeals to you, and on the unlikely chance there aren't any obvious dealbreakers to implementing this that I'm not foreseeing.

[tylermorganwall]

This could definitely be done for direct camera rays that escape the scene, but it's less clear what to do for rays that travel through a dielectric medium first: what's alpha for a ray that passes through a dielectric and then escapes to infinity? Since the ray can reflect, you can't just look to see if it escapes to infinity: e.g. looking down at a long glass plane against the ground,

the reflected rays bounce back up to infinity, but the glass should be entirely opaque. There are also paths where the ray will transmit and bounce multiple times inside the glass and eventually exit to infinity. We can also raise the glass slightly and add some attenuation and see the other side of the issue:

Here, some of the rays that hit the dielectric escape to infinity, but there is attenuation in the volume so it should be colored rather than completely transparent. I'm not sure what the alpha level should be in this case, and since there's a black background there's no color information recorded for those pixels.

There's one more issue: when an environment light is set no rays escape the scene, so this would have to be scenes with both ambient_light and environment_light set to FALSE. So this would only work on a scene with a black background.

I would say the best way to do this would be to generate an additional debug parameter like depth that returns a map of alpha values for each pixel, following whatever logic for alpha makes sense. Right now, you could currently use debug = "depth" to get a basic version of what you want (inf distance -> alpha = 0), but you'd probably want to supersample it to get soft edges (i.e. quadruple the scene resolution and then downsample) since it doesn't sample more than once.

[tylermorganwall]

This is an example for what using the depth == infinity -> alpha = 0 looks like (turn on GH dark mode to see the light):

[brodieG]

Thanks for taking the time to think about this wacky request.

That actually looks pretty good. I think loss of attenuation information and similar is okay. Obviously this can't be perfect since whatever background we do end up overlaying on (I'm thinking a solid color, so we have some leeway for subterfuge) won't be captured at all in the rendering.

In terms of reflections, I still think alpha = 0 for rays that bounce to infinity is the right answer. Imagine a perfectly reflective surface in a "black" environment e.g. on the floor. When you look down at it would be perfectly black as it's reflecting the background above the scene. If you rendered it with alpha = 0, it would also look black if you overlaid it on a black background. Technically you're looking at the background through the scene instead of bouncing up, but the result is the same. If you took that png and overlaid it on a white background, the reflection would look white, which is the same color it would look if it were rayrendered with a white background. So I think at least in that simple case it's right? And I think it generalizes to the multibounce / partial bounce? Basically, for any pixel, whatever portion goes to infinity reduces alpha by that fraction and allows the background to show through in that proportion. This happens to be equivalent to what would happen if you had reflected that fraction of rays from a sphere of the background color at infinity.

No matter what this is probably not worth much effort at all as likely the only people that care are those that both like to illustrate their websites with rayrendered images and like the subject to blend into the page background. I.e. I might be the only person to care. I was hoping there was an easy way to do it and it seems like there might be, provided we accept the limitation of solid backgrounds and attenuation, which for most of my rayrenders will be fine. And then I'll be able to have a dark theme on my website and it won't look horrible!

[tylermorganwall]

This has been added as part of 35ba1be07e56fbe59338dbd64dc1d92deba9ffe6 (v0.31.0)

[brodieG]

Oh that is soo creepy. I was just cleaning up an old laptop over the weekend and going through old mp4s and this specific issue popped in my mind, even though I'm pretty certain I did not see this notification or any other announcement until right this moment.

Thanks for implementing it! I haven't written anything on the website for a long time but maybe I'll get back to it when I'm done with the current r2c version for a change of pace. I can experiment with this then.

[brodieG]

Started to play around with this. Looks pretty good. One question is if you think it's possible to get the alpha to come through via dielectrics and metal. I.e. if something reflects off of metal into infinity, could that get recorded as transparent. Same with bending through a dielectric and ending at infinity. Obviously I don't know the details of the implementation, but it seems that for some reason the handling of a ray that goes directly to infinity is different than one that goes via reflection / refraction.

[brodieG]

For example, this:

render_scene(
  rbind(
    xz_rect(xwidth=2, zwidth=2, ),
    xz_rect(y=.1, xwidth=1, zwidth=1, material=metal()),
    sphere(y=20, material=light(intensity=20))
  ),
  lookfrom=c(.5, 3, .5),
  fov=45,
  samples=100,
  width=100, height=100,
  filename=next_file(file.base),
  transparent_background=TRUE,
)

Would result in the black part in the middle of the image being set to zero alpha. With pixels that have e.g. 50% of their rays go to infinity, alpha would be 50%. But that means tracking around an extra number for every pixel which probably makes it unworkable.

When I looked at your example originally I was too caught up in the idea that it could work that I didn't quite appreciate the dielectric still shows the original black through it.