Occlusion Material

[bobekos]

Describe the bug I use filament via the SceneForm SDK. But I suspect it is a filament bug. When I put this material on an object then on some devices (I have not yet been able to determine which ones are 100% functional and which are not.) the objects behind it are displayed in black.

This effect only affects GLTF and OBJ objects. FBX models are correctly occluded.

The question now is whether it is a filament bug or rather an ArCore bug.

This is my occluder material:

material {
    name : "Occlusion material",
    shadingModel : unlit,
    colorWrite : false,
    depthWrite : true
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor = vec4(0.0);
    }
}

To Reproduce Simply place a model with this material in front of a GLTF object. That is the result.

Expected behavior It would be correct if the object behind it were not visible. Like this:

Screenshots See "ToReproduce" and "Expected behavior" sections

Smartphone (please complete the following information):

• Device: [Not Working: Pixel 3a / OnePlus 5T] [Working: for example Samsung S7]
• OS: [All Android Versions]

Additional context The render order of the objects was set as follows: Camera -> Occlusion -> Others

If this is not a filament error I apologize. But I have the feeling that the people in the SceneForm Repo are being let down.

Maybe someone from this team can help. What surprises me is that it only does not work with GLTF and OBJ objects.

[bejado]

I'm able to fix this by giving the camera feed renderable a priority of 0 (so it is rendered first). See RenderableManager::priority.

I'm using the iOS hello-ar sample with OpenGL backend to reproduce, but I assume the same fix will work on Android. I'll need to dig into Sceneform to see where to apply the change.

Interestingly, the Metal backend doesn't seem to respect the colorWrite and depthWrite parameters, which might be a separate bug.

[romainguy]

Yes the camera needs to be renderer first for this to work.

[bejado]

@bobekos Use ArSceneView.setCameraStreamRenderPriority to set the camera stream priority through Sceneform:

The default value is 7, which forces the camera stream to render last. This is best for performance because it prevents overdraw. However, when using a material as an occluder (for example, in the augmented faces sample), this should be changed. Otherwise, the occluder will occlude the camera stream and black will be rendered.

[bobekos]

@bejado @romainguy Hi and thank you for the answers. But like i mentioned above i already set the render priority for the camera stream to 0.

Like i say the occlusion is working when i try to occlude a fbx object. When a gltf object is behind a occlusion material you see the black contour like on the screenshot above. But also not on all devices i have tested. For example the Samsung Galaxy S7 Device is working but a Pixel 3a is not.

Thats totally crazy because in the sceneform version 1.12 everything is working fine. And now i dont know if this is a filament bug or a sceneform bug. It seems to be a problem with the default gltf material which is used when a gltf object is displayed.

Without occlusion one essential point of ar is missing.

Here is the link to the sceneform bug: 925

[prideout]

Note that sceneform does not use Filament's glTF library (gltfio), instead they have their own glTF loader.

[romainguy]

The fact that it's format dependent makes me think it's likely a sceneform issue :/ (sceneform has its own fbx/gltf/obj loaders and custom materials)

[bobekos]

@romainguy Is there any chance that the sceneform team is fixing this issue? I have the feeling that the sceneform repo is no longer noticed. Are you guys still working on it? And if yes how's about issuses like this? I mean occlusion is not a small thing.

I close this ticket here. Thanks all for your time.

[vortice3D]

Hi @bobekos.

I've been following some of your issue posts on Sceneform and Filament GitHub's repositories, about how to work with occlusive not shown materials under ARCore/Sceneform/Filament.

I wonder how to set the suggested (shader) material:

material {
    name : "Occlusion material",
    shadingModel : unlit,
    colorWrite : false,
    depthWrite : true }

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor = vec4(0.0);
    }
}

...to a preloaded renderable model.

Of course, all on a Sceneform 1.16.0 basis, that is not with SFA/SFBs files but with GLTFs, through gltfio functionality.

Thanks for your time.

Best regards.

[SGTjeong]

@vortice3D Hi, To make such an material, you should use the matc tool to create matc file. Matc tool is included in Filament Library. https://github.com/google/filament/releases You can create matc file using mat file as a source.

@romainguy Hi, Thanks in advance for your effort so far.

As mentioned above, sceneform used to have their own loaders. But as far as i know, since sceneform 1.16 they now support gltf format directly with filament library.

Unfortunately I am facing the same issue with gltf object like below. Some devices are working totally fine, but some are not.

  This is the only issue remaining to us, and i desperately need any kind of crumb to fix this issue. I think @bobekos has been trying to fix this for a long time as well. Any possible help from you guys? Thanks!

[romainguy]

There's a bug open for this issue: https://github.com/google/filament/issues/2636

We don't know what's going on yet.

[vortice3D]

Hi @SGTjeong and @romainguy.

Let me thank you first for your time. By the way I'd like to apologize for using a Filament forum to post Sceneform related questions but, you know, now is almost the only way to get info about.

I'm aware of bugs and possible performance hit when using occlusion technique, considering basically the two well-known steps:

a) using occlusive-hidden material (that of course must be compiled with the aid of Filament's matc command line tool):

material {
    name : "Occlusion material",
    shadingModel : unlit,
    colorWrite : false,
    depthWrite : true
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);
        material.baseColor = vec4(0.0);
    }
}

b) change render order in the way: Camera -> Occlusive object -> Occluded objects

This said, my doubts are about HOW TO (under a Sceneform 1.16.0 basis with the aid of gltfio):

1. create a Material instance from a compiled shader/material file (the matc file obtained from the aforementioned oclusive-hidden material);
2. use/link that Material within a Renderable as call to modelRenderable.setMaterial(anyMaterial) doesn't work when using this kind of initialization:

   ModelRenderable.builder()
           .setSource( this,Uri.parse("Foo.gltf"))
           .setIsFilamentGltf(true)
           .build()
           .thenAccept(
               modelRenderable -> {
                   ARViewerActivity activity = weakActivity.get();
                   if (activity != null) {
                       renderable = modelRenderable;
                       //
                       MaterialFactory.makeTransparentWithColor(this, new Color(android.graphics.Color.RED))
                           .thenAccept(material -> {
                               renderable.setMaterial(material);
                           });
                   }
               })
               .exceptionally(
                    throwable -> {
                       Toast toast =Toast.makeText(this, "Unable to load renderable", Toast.LENGTH_LONG);
                       toast.setGravity(Gravity.CENTER, 0, 0);
                       toast.show();
                       return null;
               });

3. In a more advanced scenario, how to change material of a submesh within a GLTF/GLB file, if possible, but unluckily Renderable.getSubmeshCount is broken as you already know. I suspect it comes from a bad implementation of renderableData when a builder.isFilamentAsset is found in the (Renderable) constructor.

Even at the risk of being redundant, I wish remark that I want to link the compiled material to a Renderable under Sceneform 1.16.0, that is without the use of the former SFA/SFB mechanism (Sceneform 1.15.0 and previous, through deprecated Android Studio's plugin). Sadly, this old mechanism is the only one explained in all the articles I've found on this topic (as here, here and here).

Best regards.

[SGTjeong]

@vortice3D

1. Creating Material instance with mac file(R.raw.background is a matc file in raw folder)

   Material.builder()
            .setSource(
                context,
                R.raw.background 
            )
            .build()
            .thenAccept { material->
                 //do something with material
            }

2. I haven't called Renderable.setMaterial, so I don't know if this method is working in 1.16 or not. But Give it a try once again with the material created like above.

3. I know Renderable.getSubmeshCount returns always 0. I still can't believe why the Sceneform team hasn't implemented such an important function. If you want to access Renderable's material using Sceneform 1.16, you should use RenderableManager.getMaterialInstanceAt().

[vortice3D]

Hi @SGTjeong:

Let me first thank you for your time.

Sadly (and inexplicably), on Sceneform 1.16.0 all the Renderable's material getter/setter mechanism seems to be broken, as materialBindings property is always zero (see following Renderable class excerpt):

...
  /** Returns the material bound to the first submesh. */
  public Material getMaterial() {
    return getMaterial(0);
  }

  /** Returns the material bound to the specified submesh. */
  public Material getMaterial(int submeshIndex) {
    if (submeshIndex < materialBindings.size()) {
      return materialBindings.get(submeshIndex);
    }

    throw makeSubmeshOutOfRangeException(submeshIndex);
  }

  /** Sets the material bound to the first submesh. */
  public void setMaterial(Material material) {
    setMaterial(0, material);
  }

  /** Sets the material bound to the specified submesh. */
  public void setMaterial(int submeshIndex, Material material) {
    if (submeshIndex < materialBindings.size()) {
      materialBindings.set(submeshIndex, material);
      changeId.update();
    } else {
      throw makeSubmeshOutOfRangeException(submeshIndex);
    }
  }
...

An exception is always thrown when you try to retrieve/assign materials to/from renderables. This seems to be related with the Renderable.getSubmeshCount issue you pointed out.

This way, as you stated, I'll give a try to Filament's RenderableManager.getMaterialInstanceAt() through Sceneform's FilamentEngineWrapper class. Any advice on how to access the wrapper?

[SGTjeong]

@vortice3D

        val engine = EngineInstance.getEngine().filamentEngine
        val rm = engine?.renderableManager

        node?.renderableInstance?.filamentAsset?.let { asset ->
            for(entity in asset.entities){
                val renderable = rm?.getInstance(entity)?: 0
                if(renderable!=0 && rm!=null){
                    val mat = rm.getMaterialInstanceAt(renderable, 0)
                    mat.setParameter(..) 
                }
            }
        }

Hope, it'll help you!

[vortice3D]

Hi there @SGTjeong and thank you very much for your help.

A) I've converted your code to Java and arrived to something like this in my Activity class...

...
arFragment.setOnTapArPlaneListener(
    (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
        if (renderable == null) {
            return;
        }

    ...

     /*----------------------------Filament interfacing---------------------------------------*/
    FilamentAsset myAsset = model.getRenderableInstance().getFilamentAsset();
    Engine myEngine=EngineInstance.getEngine().getFilamentEngine();
    if(myEngine!=null){
        RenderableManager myRenderableManager=myEngine.getRenderableManager();
            if(myRenderableManager!=null){
                for (int entity: myAsset.getEntities()) {
                    if(myRenderableManager.getInstance(entity)==0){
                        continue;
                    }

                  if(myAsset.getName(entity).equals("CuboAlto_geo")){
                      MaterialInstance matInstance=myRenderableManager.getMaterialInstanceAt(myRenderableManager.getInstance(entity),0);
                      matInstance.setParameter("baseColorFactor", 1.0f, 0.0f, 0.0f, 1.0f);
                      Log.d("\n===========>\t", myAsset.getName(entity)+" material set");
                  }
              }
         }
    }
    ...
}

Of course, I'm using "baseColorFactor" here as that is a parameter used in my specific GLTF file.

That GLTF of mine has two cubes ("CuboAlto_geo" and "CuboBajo_geo") but even only setting "baseColorFactor" parameter to one of them ("CuboAlto_geo") I'm getting both shown with the red tint. How do I manage this kind of situation?

---

B) Besides the previous argumentation, is it possible to set the MaterialInstance in one go, without updating parameters one after each other? This object doesn't exhibit a public API like MaterialInstance.setMaterial or so on.

---

C) Where can I find comprehensive information on all these topics, so that I can learn at my own pace and don't have to bother others? Under my point of view, online resources on ARCore/Sceneform/Filament are not very great under a practical focus, either quantitatively or qualitatively speaking.

Thanks for your time.

[romainguy]

@vortice3D For: A. This means your two cubes are using a single material/material instance. Either you need to give one of the renderables a different instance from your code, or you need to give them different materials in your 3D package.

B. You can set the material instance of a Renderable with setMaterialInstanceAt on RenderableManager. You cannot set the Material of a MaterialInstance. Think of a MaterialInstance as a bag of values for the parameters exposed by a Material. Material == shader, MaterialInstance == values (uniforms) for that shader.

[vortice3D]

Hi @romainguy.

Well, about using MaterialInstance, I'm only following that way (instead of the preferred Material one), as a last chance to implement a custom material (to accomplish occlusive-hidden behavior in this specific case), after reading @SGTjeong warning:

If you want to access Renderable's material using Sceneform 1.16, you should use RenderableManager.getMaterialInstanceAt().

A) My own experience supports this view, because sadly the Renderable.setMaterial(myMaterial) is a no-way, due to a broken implementation of materialBindings and materialNames data structures (as stated above in this same thread, and also here and here).

B) Said that, if you think about it the MaterialInstance approach is not a solution either, because it doesn't exhibit something like a setColorWrite nor setDepthWrite methods (and I suppose they can't be reached by means of parameter settings).

As a conclusion obtained from A plus B, it seems we've reached a dead end. Any help there?

Thank you very much for your time and knowledge.

[romainguy]

MaterialInstance is the solution, a material instance is created from a Material. You can create your own material and make an instance for it.

[vortice3D]

Hi again @romainguy.

OK, but as far as I understand, getMaterialInstanceAt method is intended only for a Material that is "already" attached to a Renderable, isn't it?

How can I access a MaterialInstance directly from a Material without the need to previously attach it to a Renderable?

Best regards.

[romainguy]

You don't access a MaterialInstance from a Material, the other way around sure. But you can create a MaterialInstance from a Material using createInstance().

[vortice3D]

Hi again @romainguy.

After digging a bit deeper, in Filament docs I can see what you're trying to explain me:

Material* material = Material::Builder()
        .package((void*) BAKED_MATERIAL_PACKAGE, sizeof(BAKED_MATERIAL_PACKAGE))
        .build(*engine);
MaterialInstance* materialInstance = material->createInstance();

---

This said, I'm sure I'm missing something on all this, but under Sceneform 1.16.0, createInstance seems not be a valid public method for a Material object:

Material public APIs are very limited indeed:

---

Anyway, looking in the Material definition code (after all, that's one of the advantages of open-source, isn't it?) you can see:

@SuppressWarnings("initialization")
  private Material(MaterialInternalData materialData) {
    this.materialData = materialData;
    materialData.retain();
    if (materialData instanceof MaterialInternalDataImpl) {
      // Do the legacy thing.
      internalMaterialInstance =
          new InternalMaterialInstance(materialData.getFilamentMaterial().createInstance());
    } else {
      // Do the glTF thing.
      internalMaterialInstance = new InternalGltfMaterialInstance();
    }

    ResourceManager.getInstance()
        .getMaterialCleanupRegistry()
        .register(this, new CleanupCallback(internalMaterialInstance, materialData));
  }

So with a public access to materialData property it could be possible to access the (famous) "instance" through getFilamentMaterial().createInstance.

Sadly, that is not the case, and so maybe it's not going to be safety to make that property public on my own.

Any help?

[romainguy]

I'm talking about the Material class/API of Filament. So you do have to go through getFilamentMaterial() it seems. I am not familiar with Sceneform's APIs. Note that since Sceneform is now Open Source you can add what you need.

[vortice3D]

Hi there @romanguy and @SGTjeong :

Only for the case someone is following the conversation, wishing to follow the Sceneform way (that indeed is also a Filament way), if we review the Sceneform's Material definition (1.16.0) we find the following (package scope) method:

...
com.google.android.filament.MaterialInstance getFilamentMaterialInstance() {
    // Filament Material Instance is only set to null when it is disposed or destroyed, so any usage after that point is an internal error.
    if (!internalMaterialInstance.isValidInstance()) {
      throw new AssertionError("Filament Material Instance is null.");
    }
    return internalMaterialInstance.getInstance();
}
...

So we need only to make that method public, and then we can use a code like this for our Activity (where we are using the sceneform_opaque_colored_material instead the desired occlusive_hidden_material in order to ease the debug procedure):

...
/*----------------------------Filament--------------------------------------------*/
FilamentAsset filasset = model.getRenderableInstance().getFilamentAsset();
if (filasset!=null && filasset.getAnimator().getAnimationCount() > 0) {
    animators.add(new AnimationInstance(filasset.getAnimator(), 0, System.nanoTime()));
}
//
Engine engine=EngineInstance.getEngine().getFilamentEngine();
if(engine!=null){
    RenderableManager rm=engine.getRenderableManager();
    if(rm!=null){
        for (int entity: filasset.getEntities()) {
            if(rm.getInstance(entity)==0){
                continue;
            }

            if(filasset.getName(entity).equals("CuboAlto_geo")){
                Material.builder()
                    .setSource(
                        this,
                        R.raw.sceneform_opaque_colored_material
                    )
                    .build()
                    .thenAccept(material -> {
                        material.setFloat3("color", new Color(1.0f,0.0f,0.0f));
                        rm.setMaterialInstanceAt(rm.getInstance(entity),0,material.getFilamentMaterialInstance());
                });
            }
        }
    }
}
...

Please note the material-updating related lines with the call to the made-by-myself public getFilamentMaterialInstance API:

...
material.setFloat3("color", new Color(1.0f,0.0f,0.0f));
rm.setMaterialInstanceAt(rm.getInstance(entity),0,material.getFilamentMaterialInstance());
...

The problem is that it works only for seconds, going from the desired behavior ("high-cube" shown in red) when the app is started:

...to something showing a corrupted material like this:

...and finally crashing the app after a few more seconds.

As far as I can tell, I presume that the getFilamentMaterialInstance is being accessed concurrently from other methods of the com.google.ar.sceneform.rendering package, affecting the access to the already allocated memory in an unknown way.

P.S. This wrong behavior has been experimented with Samsung Tab S4 and Motorola Moto G6.

[SGTjeong]

Hi, @vortice3D

First of all l, i am not good at English so it's possible that i didn't understand what you are talking about.

But only looking at your code, I just wonder why you are trying to use Sceneform's Material class, and obtain Filament's MaterialInstance from it. You can create your own Filament's Material with matc file with com.google.android.filament.Material.Builder().payload().build()

As you know, Sceneform's implementation of rendering gltf is not perfect when it comes to Material. So I find it a bit risky using Sceneform's class when you do something with Material.

I am not sure of what causes the crash, but I am applying my custom material to gltf just as you are doing only except that I am creating MaterialInstance with com.google.android.filament.Material.Builder().payload().build(). It works properly.

So, I recommend you to create MaterialInstance in filament's way and check if the same error happens.

[vortice3D]

Hi @SGTjeong.

I want first to thank you very much for your time and interesting comments.

My problem with using Filament (and so on Filament's material) is the lack of a reliable info where I can learn about the different APIs. I know this render engine is open-source and no one wants to have to deal with a lot of text collection and layout, but even so a more in-depth documentation would be a plus.

About the case with com.google.android.filament.Material.Builder().payload().build(), I've seen in several examples (mostly of them in Kotlin) that its use must follow a syntax like this:

readUncompressedAsset("materials/occluder.filamat").let {
    occluderMaterial = Material.Builder().payload(it, it.remaining()).build(engine)
}

Isn't it?

My problems here come from different points:

A) I'd like to keep on using "matc materials", as with Sceneform, but the examples are talking about "filamats" here and there, loaded with the aid of readUncompressdAsset. How can I deal with this?

B) Since the (not very descriptive named) parameter "it", appears to be a file buffer, and it.remaining its size, how can I proceed with all this?

Best regards.

[SGTjeong]

Hi @vortice3D ,

A) You can use this command. matc -o ./materials/bin/YOUR_MATERIAL_NAME.matc ./materials/src/YOUR_MATERIAL_NAME.mat

B) You will be able to create ByteBuffer like below, and provide it to Material.Builder().

    try {
            val `is` = context!!.assets.open(YOUR_MATC_FILE)
            val size = `is`.available()
            val buffer = ByteArray(size)
            `is`.read(buffer)
            `is`.close()

            return ByteBuffer.wrap(buffer)
        } catch (ex: Exception) {
            ex.printStackTrace()
            return null
        }

Hope it be a help to you.

[HemanParbhakar]

@SGTjeong how to set priority to the camera and other renderables as that function is deprecated in sceneform latest version.

[ThomasGorisse]

Since nothing simce to change about the status of SceneForm since months, maybe it's time to create a fork I made one here : https://github.com/ThomasGorisse/sceneform-android-sdk

Anyone who want to help can contact me. The first goal of this fork is to make Scene Form compatible with the latests versions of AR Core and Filament.

After this, @vortice3D I invited you on this repo. Could you try to integrate your work concerning the Filament material part on Renderable maybe directly inside the Node.java class ?

If anyone wants to participate, there a lot of work to do on the glTF file usage cause right now only glb files are supported.

[romainguy]

@ThomasGorisse If you use the gltfio library from Filament you should get support for gltf/glb files right away. If you are going with a fork I would recommend trying to get rid of some of the almost empty abstractions that Sceneform has, which get in the way of manipulating Filament's types directly.

[ThomasGorisse]

@romainguy Do you suggest to simply remove SceneForm abstractions classes like Renderable and Material to directly use Filament ones?

[romainguy]

Or at least to expose the underlying Filament primitives so you don't have to duplicate APIs everywhere.

[ThomasGorisse]

I see but the only thing I regret is that those class made it a lot more easier to use than the Filament implementation. Filament is realy great and powerfull and one more time thank you for your job BUT I m not complety sure that Scene Form users need to use even 20% of the Filament features and will have to deal with a complex usage for simple things.

Anyway, before we start to work on the fork (even if I m indirectly working on it since a long time), can you confirm that SceneForm made by Google or any alternative has no future even in months ?

[romainguy]

It's about having an escape hatch. It's great that the Sceneform APIs are easy to use, but often folks wanted to do something that wasn't exposed but that Filament could do underneath. It's just about offering the ability to jump one level down to do more advanced things when needed.

I don't work on Sceneform/Daydream so I unfortunately cannot comment on whatever plans they might have :(

[ThomasGorisse]

I hope, things will move fast on the Scene Form team communication. I feel like I'm living again what happened with ActionBarSherlock : Something that was used in every app by every developers but not made by Google.

Thanks for your time

[romainguy]

I wish I could be more helpful. Feel free to ask questions and file features requests that are Filament related though!

[ThomasGorisse]

The problem is that I have no problem with Filament since it's working great but I have with Scene Form which has been abonned in the middle of the 1.6.0 Filament migration leaving no adrdress when the new adoptors need help ;-)