Combining gtlf and .filamat together. No Occulsion

[HemanParbhakar]

Hello I am using the sceneform repo 1.17 in which they have used filamat to render stuff. I want to perform occlusion for which :

• I have create a mat file:

material { name : "Occlusion material", shadingModel : unlit, colorWrite : false, depthWrite : true }

fragment { void material(inout MaterialInputs material) { prepareMaterial(material); material.baseColor = vec4(0.0); } }

• I have compiled it to .filamat file using :

matc -p mobile -a opengl -o work.filamat work.mat

• I have set a renderable manager to priority of 1 private void setOcclusionElement() { ByteBuffer payload = readUncompressedAsset(); Engine engine = EngineInstance.getEngine().getFilamentEngine(); MaterialInstance materialInstance = new com.google.android.filament.Material.Builder().payload(payload.rewind(), payload.remaining()).build(engine).getDefaultInstance(); int renderable = EntityManager.get().create(); new RenderableManager.Builder(1).priority(1).material(0, materialInstance).build(engine, renderable); engine.createScene().addEntity(renderable); }

private ByteBuffer readUncompressedAsset() { ByteBuffer dst = null; try { InputStream is = getAssets().open("work.filamat"); int size = is.available(); byte[] buffer = new byte[size]; is.read(buffer); is.close(); dst = ByteBuffer.wrap(buffer); } catch (Exception ex) { Log.e("Error", ex.getLocalizedMessage()); } return dst; }

ModelRenderable.builder() .setSource( this, Uri.parse( "https://storage.googleapis.com/ar-answers-in-search-models/static/Tiger/model.glb")) .setIsFilamentGltf(true) .build() .thenAccept( modelRenderable -> { GltfActivity activity = weakActivity.get(); if (activity != null) { activity.renderable = modelRenderable; } }) .exceptionally( throwable -> { Toast toast = Toast.makeText(this, "Unable to load Tiger renderable", Toast.LENGTH_LONG); toast.setGravity(Gravity.CENTER, 0, 0); toast.show(); return null; });

` arFragment.setOnTapArPlaneListener( (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> { if (renderable == null) { return; } setOcclusionElement();

                // Create the Anchor.
                Anchor anchor = hitResult.createAnchor();
                AnchorNode anchorNode = new AnchorNode(anchor);
                anchorNode.setParent(arFragment.getArSceneView().getScene());

                // Create the transformable model and add it to the anchor.
                TransformableNode model = new TransformableNode(arFragment.getTransformationSystem());
                model.setParent(anchorNode);
                model.setRenderable(renderable);
                model.select();

                FilamentAsset filamentAsset = model.getRenderableInstance().getFilamentAsset();
                if (filamentAsset.getAnimator().getAnimationCount() > 0) {
                    animators.add(new AnimationInstance(filamentAsset.getAnimator(), 0, System.nanoTime()));
                }

                Color color = colors.get(nextColor);
                nextColor++;
                for (int i = 0; i < renderable.getSubmeshCount(); ++i) {
                    Material material = renderable.getMaterial(i);
                    material.setFloat4("baseColorFactor", color);
                }

                Node tigerTitleNode = new Node();
                tigerTitleNode.setParent(model);
                tigerTitleNode.setEnabled(false);
                tigerTitleNode.setLocalPosition(new Vector3(0.0f, 1.0f, 0.0f));
                ViewRenderable.builder()
                        .setView(this, R.layout.tiger_card_view)
                        .build()
                        .thenAccept(
                                (renderable) -> {
                                    tigerTitleNode.setRenderable(renderable);
                                    tigerTitleNode.setEnabled(true);
                                })
                        .exceptionally(
                                (throwable) -> {
                                    throw new AssertionError("Could not load card view.", throwable);
                                }
                        );
            });

` In the above methods I have read the .filamat file as byte buffer . Have set a renderable manager with a priority of 1. Have added a gltf object on tapping of ar. The camera priority is set

https://github.com/google-ar/sceneform-android-sdk/blob/b33fa25987561e5bf2129a25fbce6ad3b43ac4ee/sceneformsrc/sceneform/src/main/java/com/google/ar/sceneform/ArSceneView.java#L713

private void initializeCameraStream() { cameraTextureId = GLHelper.createCameraTexture(); Renderer renderer = Preconditions.checkNotNull(getRenderer()); cameraStream = new CameraStream(cameraTextureId, renderer); cameraStream.setRenderPriority(0); }

So now all the conditions are met Mat file with depthWrite:true, The priority must be: Camera > Occluder renderables > Others renderables

Why it is still not occluding

[romainguy]

Your renderable does not have any geometry:

new RenderableManager.Builder(1)
    .priority(1)
    .material(0, materialInstance)
    .build(engine, renderable);

It's therefore not rendering anything. You also add it to a scene that you don't use anywhere:

engine.createScene().addEntity(renderable);

The scene needs to be given to a View to be rendered, you just discard it.

[HemanParbhakar]

i have done that also but the tiger is still not occulding. private void setOcclusionElement() { ByteBuffer payload = readUncompressedAsset(); Engine engine = EngineInstance.getEngine().getFilamentEngine(); MaterialInstance materialInstance = new com.google.android.filament.Material.Builder().payload(payload.rewind(), payload.remaining()).build(engine).getDefaultInstance(); int renderable = EntityManager.get().create(); new RenderableManager.Builder(1).priority(1).material(0, materialInstance).build(engine, renderable); engine.createScene().addEntity(renderable); }

[romainguy]

In the example above your occluder has no geometry and is not added to the same scene as the tiger. There's no occlusion possible with this setup.

[HemanParbhakar]

but why so i have done as you said in https://github.com/google/filament/issues/3100

[HemanParbhakar]

Geometry is already given as

private ByteBuffer readUncompressedAsset() { ByteBuffer dst = null; try { InputStream is = getAssets().open("work.filamat"); int size = is.available(); byte[] buffer = new byte[size]; is.read(buffer); is.close(); dst = ByteBuffer.wrap(buffer); } catch (Exception ex) { Log.e("Error", ex.getLocalizedMessage()); } return dst; }

private void setOcclusionElement() { ByteBuffer payload = readUncompressedAsset(); Engine engine = EngineInstance.getEngine().getFilamentEngine(); MaterialInstance materialInstance = new com.google.android.filament.Material.Builder().payload(payload.rewind(),payload.remaining()).build(engine).getDefaultInstance(); int renderable = EntityManager.get().create(); new RenderableManager.Builder(1).priority(1).material(0, materialInstance).build(engine, renderable); engine.createScene().addEntity(renderable); }

It is already done . You have missed to check it

[romainguy]

No, there is no geometry passed to the renderable. Here is your code:

new RenderableManager.Builder(1)
    .priority(1)
    .material(0, materialInstance)
    .build(engine, renderable);

You are creating a renderable that only contains a material. There are no triangles in that renderable. In addition, like I said earlier, you are adding that renderable to a newly created scene that you discard:

engine.createScene().addEntity(renderable);

You call createScene() without keeping a reference to that Scene. That scene won't be rendered if you don't give it to a View for rendering.

[HemanParbhakar]

@romainguy the payload is reading the filamat file and that is used later in the payload and created a default instance and used later when building RenderableManager ByteBuffer payload = readUncompressedAsset(); MaterialInstance materialInstance = new com.google.android.filament.Material.Builder().payload(payload.rewind(),payload.remaining()).build(engine).getDefaultInstance()

[romainguy]

I understand your code, but you are only reading the material file, you are not creating any geometry to render with. The material needs to be applied on geometry (a 3D model) to have an effect. That's the part you are missing. The problem is not loading the material in your case. Please read my answers above:

• You need geometry in your renderable (the triangles forming a 3D mesh)
• The renderable needs to be in the same scene as the tiger, not a new scene

[HemanParbhakar]

@romainguy I only want to render the tiger and make it occuled.

[romainguy]

I understand, but what do you want to occlude it with? Your occluder needs to have a "shape", that's what the geometry is for.

[HemanParbhakar]

I just wanna hide the tigers behind the walls. Basically i want to hide the virtual object behind the physical object so RenderableManager.Builder(1) // Overall bounding box of the renderable .boundingBox(Box(0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f)) // Sets the mesh data of the first primitive, 6 faces of 6 indices each .geometry(0, PrimitiveType.TRIANGLES, vertexBuffer, indexBuffer, 0, 6 * 6) // Sets the material of the first primitive .material(0, materialInstance) .build(engine, renderable)

    // Add the entity to the scene to render it
    scene.addEntity(renderable)

https://github.com/google/filament/blob/main/android/samples/sample-hello-camera/src/main/java/com/google/android/filament/hellocam/MainActivity.kt#L134 So like this i need to add it.

If you are saying the geometry i need to use. But the enviorment make change everytime so what about that ?

[HemanParbhakar]

@romainguy one more thing. Do filament support anchor cloud or local anchor like placing AR objects on the predefined values.

[romainguy]

Anchors are not a concept that needs to be in Filament. ARCore will provide you with anchors and you use those anchors to set transforms on Filament's renderables. Filament is only a rendering engine.

[HemanParbhakar]

@romainguy why there is no proper documentation regarding this. There is only regarding filament graphics and materials.

[HemanParbhakar]

what is way to add the glb into scene

try { InputStream is = getAssets().open("models.glb"); int size = is.available(); byte[] buffer = new byte[size]; is.read(buffer); is.close(); ByteBuffer dst = ByteBuffer.wrap(buffer); modelViewer.loadModelGlb(dst); modelViewer.transformToUnitCube(new Float3(0, 0, -4)); FilamentAsset filasset = modelViewer.getRenderableInstance().getFilamentAsset(); if (filasset != null && filasset.getAnimator().getAnimationCount() > 0) { animators.add(new AnimationInstance(filasset.getAnimator(), 0, System.nanoTime())); } if (engine != null) { RenderableManager rm = engine.getRenderableManager(); if (rm != null) { for (int entity : filasset.getEntities()) { if(rm.getInstance(entity)==0){ continue; }

                }
            }
        }
    } catch (Exception ex) {
        ex.printStackTrace();
    }

Like i have iterated over the renderable manager but how to get the material instance so that i can set to the RenderableManager and added to the scene.

[HemanParbhakar]

Anchors are not a concept that needs to be in Filament. ARCore will provide you with anchors and you use those anchors to set transforms on Filament's renderables. Filament is only a rendering engine.

as you said transforms need to be used. But Filament has not its own Renderable. and Renderable Manager would need a material instance that would be from the glb/gltf. then material has no such properties of updating the transform. So how to handle that ?

[romainguy]

Transforms are not part of the materials, they are a transform component set by the transform manager on an entity (a renderable in your case).

[HemanParbhakar]

Got there is TransFormManager that will work. But how to get the material instance of glb