Resolve alpha depth issues automatically

[ngokevin]

Issue by jcarpenter Friday Nov 20, 2015 at 23:32 GMT Originally opened as https://github.com/aframevr/aframe-core/issues/498

---

It will be common to superimpose images w/ alpha channels. There is a currently a footgun where order of declaration in the DOM must be carefully managed, or transparent elements will cull each other.

For example, in the AnimeUI test scene in A-Frame, I have a series of transparent images arranged on the z-axis. I must declare them in order of depth, furthest from camera to closest, or I get blending problems.

<vr-image src="images/schematic1.png" width="7" height="7" position="0 0 -6"></vr-image>
<vr-image src="images/ring2.png" width="3" height="3" position="0 0 -2"></vr-image>
<vr-image src="images/ring1.png" width="0.5" height="0.5" position="0 0 0"></vr-image>

This is what it looks like when the user takes the correct steps:

This is problematic because:

• The failure state is obvious and ugly
• The cause will not be obvious to users
• The resolution is sometimes not feasible. The point of A-Frame is grouping elements to fit your scene layout and mental model. The current resolution forces you to group things by z-axis.

We should solve this automatically for users.

[ngokevin]

Comment by jcarpenter Friday Nov 20, 2015 at 23:34 GMT

---

I believe Jaume refers to this issue in a recent tweet re: OIT (order independent transparency):

https://twitter.com/mrdoob/status/667835175108112384

Paging @caseyyee who's dealt with this before. Perhaps @fernandojsg also has some suggestions.

[ngokevin]

Comment by caseyyee Saturday Nov 21, 2015 at 00:19 GMT

---

I've had to fiddle with a bunch of different properties in the past. I really don't fully understand how each one works under the hood, but if re-arranging the order in which objects are added to the scene doesn't work, i've used some of these properties in different combinations to get things working:

Mesh.renderOrder Controls the order in which objects are rendered.

Material.depthTest=false Whether to have depth test enabled when rendering this material. Default is true.

Here's a good fiddle I found that demonstrates the combo: http://jsfiddle.net/sg02e5sm/2/

Material.alphaTest Sometimes the alpha on the texture itself is problematic. Adjusting this value can make it work properly. It does clip data sometimes though if you are not careful.

Renderer.sortObjects Defines whether the renderer should sort objects. Default is true.

Each of these have different impacts on perf as well (especially forcing sort order).

As per @jcarpenter, The user shouldn't have to deal with any of this. I think we could probably be intelligent with how objects are sorted in the scene to make it work for most people.

[ngokevin]

Comment by jcarpenter Saturday Nov 21, 2015 at 02:18 GMT

---

More from Doob: https://twitter.com/mrdoob/status/667850994068422656

I fear OIT cant be easily solved by mere mortals like us in the application layer. GPUs should solve that for us...

[ngokevin]

Comment by jcarpenter Saturday Nov 21, 2015 at 02:18 GMT

---

Adding @kearwood to thread also...

[ngokevin]

Comment by fernandojsg Saturday Nov 21, 2015 at 09:57 GMT

---

Yes as mrdoob says it's not an easy problem and there're a lot of papers about how to do a real OIT. @caseyyee pointed some properties that can be use for simple scenes and mainly for billboards as the one used on particles that they're just simple planes. The problem is when the objects intersect between them, just as simple as like two boxes colliding. If two boxes intersect each other you can't fix it by change the rendering order, in that case you should split the faces and do a sorting for faces not objects. In this case the sorting is a huge bottleneck. One step further is when some faces intersect each other in this case, the most general one, you need to sort by fragments not just even by faces, so the result is a huge performance problem. So as mrdoob pointed the only easy solution could be that the GPU would take care of that for the most general case, meanwhile I believe we could just rely on this kind of tricks and simple properties to finetune our scene :disappointed:

[ngokevin]

Comment by jcarpenter Thursday Nov 26, 2015 at 04:06 GMT

---

Just spent 20 minutes trying to figure out why animating opacity on a foreground element was making all other images in the scene disappear. It’s because they were further away in z-space, and I forgot to take into account this issue. I definitely foresee some huge user confusion on this one...

[ngokevin]

Comment by jcarpenter Sunday Nov 29, 2015 at 22:28 GMT

---

Sent email to Games mailing list about this...

[kennardconsulting]

Test case for 0.5.0:

The yellow square with opacity: 1 works fine. The yellow square with partial opacity does not appear unless the DOM elements are re-ordered:

<html>
<head>
    <script src="aframe-v0.5.0.min.js"></script>
</head>
<body style="background-color: black">
    <a-scene>
        <a-plane material="color: red; opacity: 0.5" position="0 1 -2"></a-plane>
        <a-plane material="color: yellow;" position="-0.6 1.6 -2.1"></a-plane>
        <a-plane material="color: yellow; opacity: 0.5" position="0.6 1.6 -2.1"></a-plane>
        <!-- Move red plane here and it works -->
    </a-scene>
</body>
</html>

[MagicIndustries]

I have an issue where a fadeIn / fadeOut of a simple plane in front of the camera is making other elements with transparency disappear. Is this the same issue?

I'm adding the fade plane dynamically, so should I be trying to add it to the top or the bottom of the DOM to try to beat this?

I should note the fade plane ends up being a child of the camera currently.

And since this is something I'm working on at work, if there's anything I can do to help move the final solution forward just let me know. I have no idea where this would start though :)

[ngokevin]

material="alphaTest: 0.5" seems to magically help in most cases.

[jfaust]

Is there a reason that the standard solution to this (sorting transparent objects based on distance to the camera), is actively disabled by AFrame? Admittedly it doesn't always work but in most situations it works way better than manually controlling the render order. Especially in head-tracked VR, where the user's head can be anywhere.

At least having the option to turn it on would be great.

I'm referring specifically to a-scene.js:486

renderer.sortObjects = false;

[ngokevin]

I'm not sure why that's disabled. @dmarcos, aware of any history?

[dmarcos]

As far as I remember it was disabled to favor manual control. We can revisit. I have not dealt much with transparent objects so no familliar with all the problematic of automatic sorting.

[jfaust]

Consider turning it on by default with an option to turn it off. Having it off works well for 3DOF experiences with no movement, or very fixed movement, but the addition of 6DOF or even teleport will break most manually-ordered objects.

The case where sorting really breaks down is with intersecting (or self-intersecting) objects, since the sort is per object, not per triangle (though in the wider world where compute shaders are available, sorting per-triangle is sometimes used). To support manual ordering of some objects for those cases you could then expose threejs's Object3D.renderOrder property if it's not already.

The accepted answer here has a good explanation of the problems and potential solutions (including one of the replies which talks about the render queues): http://answers.unity3d.com/questions/609021/how-to-fix-transparent-rendering-problem.html

[donmccurdy]

+1 for defaulting to true with an option to disable.

[trusktr]

Can you post an example or screenshot of what happens when the order is not by Z index, to see the ugly effect?

[donmccurdy]

Here is an example: https://gltf-viewer.donmccurdy.com/#model=https://raw.githubusercontent.com/KhronosGroup/glTF-Blender-Exporter/master/polly/project_polly.gltf

^Large file; best viewed on desktop.

angle 1	angle 2

As you rotate the scene around, watch the glass display sign at the front of the scene. The "Polly" text will appear and disappear from the sign, depending on camera orientation, because of transparency sort issues. This type of problem cannot be trivially solved in WebGL... the proper solution here would be for the artist to split the mesh of the glass casing in two parts, so that transparency sort will resolve correctly.

[trusktr]

Interesting. Thanks. So sorting order for transparency is the order in which the nodes are found in the scene tree? Maybe it can be configurable with an attribute, like sort-order="3" to not have to worry about tree order.

[trusktr]

Does Three.js have any such feature? Or does it too rely on tree order?

[donmccurdy]

Scene tree order isn't used, the order is:

1. draw opaque objects, in order from furthest away to nearest.
2. draw transparent objects, in order from furthest away to nearest.

Three.js does expose an object.renderOrder property to override sort order. You can also turn off the depth sort, in which case order is probably based on the scene tree but I'm not sure. Related reading: https://www.khronos.org/opengl/wiki/Transparency_Sorting

Usually three.js's defaults are what you want. A-Frame does not currently expose an HTML attribute to override sort order, but this is easy to implement.

[jfaust]

Tree order is (I assume) used in the case where renderer.sortObjects is set to false (like AFrame currently does). Unless you're controlling the camera viewpoint very carefully this is almost never what you want as a global setting.

[adelespinasse]

Is there some reason why the renderer.sortObjects = false; line still hasn't been removed? Seems like such a simple fix that would cover 95% of cases, and there's a reason why the default value is true in three.js. I'll submit a pull request if it's just because no one has got around to it.

For anyone looking for a quick workaround, see this answer.

[donmccurdy]

I do still say +1 for removing that line. Even better (IMO) would be to create a renderer scene component exposing a few more advanced options:

<a-scene renderer="sortObjects: true;
                   gammaOutput: true;
                   physicallyCorrectLights: true">
  ...
</a-scene>

^The other two properties I suggest are required for "technically correct" glTF rendering. These can be done in user-land components, but to change gammaOutput you have to recompile every shader in the scene, so it's definitely preferable to have that set correctly during initialization.

[wmurphyrd]

@adelespinasse I tried using sort objects once after seeing it in this thread and the results were very poor in VR. Render order would change as I turned my head, causing the order of objects to flicker back and forth. Have you had better results?

[donmccurdy]

Suggested a fix in #3424, and we could leave the default sortObjects=false while still allowing others to override it. For what it's worth, since I don't see it mentioned elsewhere in the thread, it is easy to do a component patching this as needed:

AFRAME.registerComponent('my-renderer-settings', {
  init: function () {
    var sceneEl = this.el;
    sceneEl.renderer.sortObjects = true;
    sceneEl.renderer.gammaOutput = true;
  }
});

[adelespinasse]

@wmurphyrd It's worked for me so far, but I've only tried it with very simple toy scenes (like, 3 non-overlapping convex translucent objects). Did you have concave or overlapping objects or anything?

I don't know the details of how the sorting works... are objects always sorted for every frame, or is it done opportunistically when there's time or something? And are objects sorted separately for each eye?

[tommensink]

While this does work better for transparent elements added through AFrame, it does not work for multiple transparent objects within a glTF object. I guess the DOM of that object needs to be traversed to set each node correctly

[donmccurdy]

Note that the option of enabling sorting is there in 0.8.2+:

<a-scene renderer="sortObjects: true;" > ... </a-scene>

This can resolve many issues, at the expense of being view-dependent – sort order may change as the camera moves. Also, always keep transparent: false; alphaTest: 0.5 (or some other alphaTest value) any time your texture has only fully-opaque and fully-transparent areas, such as a cutout leaf/branch pattern.

[tommensink]

Great, I did not know that it was included already (used your code above). But material settings do not have per default affect on glTF as you know, so the other settings do not have any effect. (<a-entity gltf-part could use that, but no luck with that either).

But I think you are right, it should not depend on the camera position. In one case I have two transparent spheres on the same position, one a little smaller than the other. They have snowflakes and contra-moving animations so that a snowglass effect is achieved. It is quite a nice testcase to get this working. https://raw.githubusercontent.com/gltfs/any/master/xmas_2018/art_snow/art_snow.glb

Babylon.js gltf viewers have no problem with it. Three.js viewers, like your own @donmccurdy only show the outer sphere. You can test it yourself if you like :-)

If we cannot solve transparency with three.js, maybe we should switch to babylon.js for aframe? I mean, what is the future of this all without transparancy?

[donmccurdy]

The transparent/alphaTest settings can be baked into a glTF model; e.g. when exporting from Blender 2.8 you’d use the “Alpha Clip” blend mode.

Semi-transparency is a complex in any realtime renderer... note that Unity’s deferred pipeline disallows it entirely. If BabylonJS can handle this case then presumably there are fixes we could make in aframe or threejs, but you should assume BablyonJS will have its own, different limitations with transparency.

While that’s inconvenient, there’s a long history of working around this in games by various means. Using alpha clipping to avoid semi-transparency is a common one. Additive blending is another, which might be appropriate for snowflakes.