Mesh highlighting

[chun-baoluo]

Hi! Mesh highlighting doesn't seem to work for me. I add a highlight layer and the picture comes out almost completely black. Same behaviour on both Android and iOS. Also, there is no way to set stencil option for EngineView (maybe that's why highlighting doesn't work properly).

[CedricGuillemet]

I've done the plumbing to support stencil func in BabylonNative. But I faced the same issue for the shadows: a post process pass is performed to blur the result. Because of inverted Y in render target pass, highlight is also Y inverted.

[chun-baoluo]

@CedricGuillemet I built BabylonNative with recent stencil commit on my machine, replaced files in react-native folder with content of Assembled folder, updated @babylonjs/core package to alpha-29. App launches without a problem, there is no freezing like alpha-29 did prior file replacement. However highlight layer still doesn't seem to work. I enable stencil through engine.setStencilBuffer(true) and the picture comes out almost the same as in my first post, except colouring pattern now inverted. Am I doing something wrong or there are more changes to be done on your side in order for highlight layer to work?

[ryantrem]

@rarattay you got mesh highlighting working recently right? Any insights here?

[rarattay]

I was able to get mesh highlighting via a HighlightLayer to work in the Babylon Native and Babylon React Native playgrounds once @CedricGuillemet 's initial stencil buffer support PR went in (https://github.com/BabylonJS/BabylonNative/commit/30a574bf1f337444388c87fa9d5bd7f4542f9517).

To get this working all I did was pull latest master of this repo (Babylon React Native) and then updated the BabylonNative submodule to latest master (inclusive of the commit linked above). Then to test out adding a highlight layer I added this right after the default model is imported in playground's App.tsx:

result.meshes.forEach(mesh => { var hl = new HighlightLayer("hl1", scene); hl.addMesh(mesh as Mesh, Color3.White());
});

[ryantrem]

I believe @chrisfromwork will be updating BabylonReactNative to the latest BabylonNative and publishing a new NPM package in the next few days.

[CedricGuillemet]

I'm still debugging highlighting + XR. There a 1 frame lag. I hope to get it fixed in the coming days.

[chun-baoluo]

@ryantrem I checked new package version (0.4.0-alpha.23) and got the same results - everything is pitch black except highlighted meshes which are completely covered in highlight color overlay-style. Then I tried testing highlight layer using App.tsx code from your Playground app and everything again renders pitch black after enabling highlight layer and adding mesh to it.

[CedricGuillemet]

Tested ok with AR thanks to this PR : https://github.com/BabylonJS/BabylonNative/pull/819

[ryantrem]

@star-collector will you commit your test code in a branch, push to your fork, and share a link so we can investigate a repro?

[chun-baoluo]

@star-collector will you commit your test code in a branch, push to your fork, and share a link so we can investigate a repro?

Yes, I can. But first, can you tell me if I enable stencil the right way for BabylonReactNative?

engine.setStencilBuffer(true);

I do this right before I add HighlightLayer. Is this correct?

[CedricGuillemet]

You don't have to change anything with the stencil to get a highlighting layer. Basically., you just need to create one and add mesh to it:

var box = BABYLON.Mesh.CreateBox("box1", 0.2);
var hl = new BABYLON.HighlightLayer("hl1", scene);
hl.addMesh(box, BABYLON.Color3.Green());

[chun-baoluo]

@CedricGuillemet @ryantrem

Ok, I tested again and it seems like the problem persist only on iOS. For some reason, I didn't test HighlightLayer layer on Android before. It works on this platform:

However, I still get this on iOS:

And for some reason I still get warning about stencil in console:

I'll provide sample app if needed. I use latest npm packages of @babylonjs/core and @babylonjs/react-native.

[CedricGuillemet]

I can repro on iOS with latest master. I'm investigating.

[CedricGuillemet]

I've found a workaround. It looks like MSAA + stencil do not work on iOS device (works on simulator and on MacOS, so might not be a Metal issue. Maybe a driver one). To disable MSAA for Metal, remove BGFX_RESET_MSAA_X4 on this line : https://github.com/BabylonJS/BabylonNative/blob/954a6007b1b5ef6c4f45297e44e486fbad7dd177/Core/Graphics/Source/Apple/Babylon/GraphicsPlatformImpl.h#L7

I need more time to do a smaller repro and figure out the real problem.

[ryantrem]

Should we just disable MSAA on iOS for now, especially since it is disabled on Android? What is the tangible visual impact? How does it affect perf?

[chun-baoluo]

I noticed another highlighting bug. If a highlighted object is only partially inside a camera, then the opposite side of the camera starts glowing as well. Problem exist on both Android and iOS.

Here this bug is visible on both shelf and TV. Left side is glowing, however there are no actual objects to glow. Also, right side of the camera and character herself are partially glowing as well. As if she is behind the highlight and not in front.

[CedricGuillemet]

I was able to find a repro and it seems related to our rendering lib. An issue has been open:

https://github.com/bkaradzic/bgfx/issues/2620

I continue the investigations.

[bghgary]

Should we disable MSAA on iOS to work around this for now? MSAA is already disabled on Android.

[CedricGuillemet]

Should we disable MSAA on iOS to work around this for now? MSAA is already disabled on Android.

Enabling MSAA on Android will crash whereas MSAA on iOS will "only" make highlighting rendering glitched. I don't know if it's better to disable it or only warn/alert when user face the problem. And user will disable MSAA themselves. It doesn't seem like many users are using HL on iOS for now.

[CedricGuillemet]

Closing this issue. https://github.com/BabylonJS/BabylonNative/issues/988 has been open for centralizing issues with MSAA.