Sample app demonstrating ARCore/Filament integration

[tellypresence]

Now that SceneForm has been retired, would be very useful to have a basic sample app demonstrating ARCore/Filament direct integration

As suggested here https://github.com/google/filament/issues/1551#issuecomment-524508659

[tellypresence]

Thank you for assigning this. In the meantime and in the spirit of constructive progress, I managed to make a "mash up" of three projects:

• sample-transparent-view (filament android sample)
• sample-image-based-lighting (filament android sample)
• hello_ar_java (ARCore android sample)

The result is the "shader ball" object spinning in front of the working AR sample. While a promising sign-o-life, there are of course several issues with this crude and naive approach leading to some questions/request for advice:

• The IBL sample uses filamesh intermediate format rather than gltf/glb; how to load gltf/glb directly? (not sure if this feature is exposed as it looks like filament ModelViewer object may wrap gltf loading)
• The hello_ar sample uses "bare metal" OpenGL throughout, including for rendering the camera preview; what are best practices for integrating the GlSurfaceView with filament's surfaceView? (IIRC elsewhere there is mention of sharing GL context with filament but documentation seems to be more geared to PBR features right now)
• How to apply the usual MVP matrices from ARCore tracker to the filament model?
• What are best practices for loading multiple models (crucial for moving beyond "toy" project stage)
• (And this is more of a luxury at this early stage) How to wire up filament's IBL with the environmental lighting model provided from ARCore?

What I really would like to be able to do is convert filament sample sample-gltf-viewer to work with ARCore, but that seems like it may be difficult since filament ModelViewer appears to hide a lot of required state, and the integrated skybox may make it difficult to use the transparent texture technique.

[romainguy]

I’ll respond in more details tomorrow but:

• You don’t need ModelViewer, it’s just a convenience. gltfio lets you load gltf or glb files with AssetLoader
• Do not use GlSurfaceView. Everything the ARCore sample does can be done through Filament
• You can check the source of SceneForm to see how to use ARCore’s dynamic lighting and how to share the GL context
• ARCore transforms can be applied to filament renderables and cameras. There’s nothing special there
• Loading multiple models is the same as loading a single model, using gltfio

All the APIs used by SceneForm are exposed in Filament. You should be able to find all the answers there.

[tellypresence]

The more details the better:smile:, thank you

[romainguy]

Some more info:

• You can give Filament the camera stream via a Stream object that you then set on a Texture using setExternalStream. See CameraStream and ExternalTexture. The only "trick" is that you need to use a samplerExternal in your material to read from such textures.
• To apply light estimation from ARCore, see Sceneform's LightProbe. In particular check the call to generatePrefilterMipmap() which is where ARCore's IBL is passed to Filament for processing.
• For transforms, check AnchorNode. Turning the ARCore pose into a matrix happens in Node.

[tellypresence]

Thank you for your advice, I'm trying to use the camera feed as background using the stream approach you suggested; my starting point was to modify the sample-hello-camera sample.

Sorry for a basic question -- I tried to change the cube mesh to fit the screen with no success; AFAICT tweaking the params of the boundingBox block in the RenderableManager builder has no effect; the cube is always the same size/proportions even when halfExtentX and halfExtentY are different values. Any specific pointers/sample code on how to set up the mesh to fit the screen, or better still a simple 2-tri quad that does the same thing?

[tellypresence]

On the subject of modifying the samples, I was hoping to also display both the spinning triangle from the sample-transparent-view and the oscillating cube from sample-hello-camera simultaneously; unfortunately the triangle is in the wrong location to use anything but the ortho camera projection; trying to use perspective or FOV causes the triangle to disappear because its transform isn't compatible with the camera. Any pointers on how to get that 2D spinning triangle to cooperate with standard perspective camera? Again I tried modifying the boundingBox params (i.e. shifting Z to try to get it into viewing frustum) with no luck

[romainguy]

The bounding box is just that, a bounding box. It doesn't affect the geometry. It's used for culling. If you want a fullscreen mesh you can use just one triangle. In its material set the vertex domain to be device. Now the entire screen can be covered using coordinates between -1 and 1. So for a simple triangle just have it use the vertices -1,-1,0/3,-1,0/-1,3,0. It will cover the entire screen.

[tellypresence]

What's the proper way to set (read-only property) vertexDomain? Something like

material.setDefaultParameter("vertexDomain", Material.VertexDomain.DEVICE.ordinal)

or

materialInstance?.setParameter("vertexDomain", Material.VertexDomain.DEVICE.ordinal)

(AFAICT none of the samples use that)

[romainguy]

See the documentation: https://google.github.io/filament/Materials.html#materialdefinitions/materialblock/vertexandattributes:vertexdomain

[MihailovDev]

Hey, this is probably not a question for here but ... I want to implement augmented images with ar core. I want to scan a coloring book and than place a 3d object on top of the drawing. Most of my research shows example of sceneform for augmented images. I am really a beginner in ARCore and have not implemeted more than the basic examples and codelabs. I would like an advice in the sense of where should i begin and some key search terms that i ought to be looking for. Thanks

[Drjacky]

@tellypresence I have the exact need you mentioned. Do you have any working sample?

[tellypresence]

@Drjacky no, but hopefully if enough ex-SceneForm users express interest, the proposed sample app may eventually be provided

[Drjacky]

@tellypresence I've found this: https://github.com/google-ar/sceneform-android-sdk/tree/master/samples/gltf.

Hope it helps you too.

[MihailovDev]

I have not made any progress.

On Tue, 28 Jul 2020 at 09:48, Hossein Abbasi notifications@github.com wrote:

@tellypresence https://github.com/tellypresence I've found this: https://github.com/google-ar/sceneform-android-sdk/tree/master/samples/gltf .

Hope it helps you too.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/google/filament/issues/2646#issuecomment-664837647, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKMQWQ34S3P7NOMTU26SZRTR5Z7GRANCNFSM4NTFSUCQ .

-- Георги Михаилов

Главен извршен директор

Culture Technologies LLC +389 78 368 313 <+389+78+368+313> mihailovdev@gmail.com www.razgovor.mk https://www.flow.com.mk/ Sutjeska 11, Stip

[tellypresence]

@Drjacky Modified version of that gltf sample attached to issue 1046

(Duplicated instead of referenced/instanced 3D models leads to OOM crashes and is a showstopper for us; this issue did not affect legacy SceneForm and is the reason we're stuck on sceneForm v1.15.0. IIUC GL instancing should be ready in future filament release)

[wintersim]

Hello, I am currently trying to make an ArCore app and since SceneForm is now (officially?) deprecated and my OpenGL knowledge is not enough for pure ArCore I was told to take a look at Filament. Now the problem is that I don't really know were to start.. So I just wanted to say that an ArCore example app (maybe something like the Hello AR example) would be very appreciated.

[DoctorB]

Some more info:

• You can give Filament the camera stream via a Stream object that you then set on a Texture using setExternalStream. See CameraStream and ExternalTexture. The only "trick" is that you need to use a samplerExternal in your material to read from such textures.
• To apply light estimation from ARCore, see Sceneform's LightProbe. In particular check the call to generatePrefilterMipmap() which is where ARCore's IBL is passed to Filament for processing.
• For transforms, check AnchorNode. Turning the ARCore pose into a matrix happens in Node.

Hello @romainguy, I was able thanks to your advice to have a working camera feed. In the CameraStream there is a transformDisplayUvCoords method that I imagine performs at least one scale and one rotation operation. My camera's feed is now clearly not scaled or rotated correctly. The shader I'm using is the same one used by the sceneform for the camera. Do you have any suggestions for correctly rotating and rescaling the texture without having the call to transformDisplayUvCoords available?

[Drjacky]

Hello, I am currently trying to make an ArCore app and since SceneForm is now (officially?) deprecated and my OpenGL knowledge is not enough for pure ArCore I was told to take a look at Filament. Now the problem is that I don't really know were to start.. So I just wanted to say that an ArCore example app (maybe something like the Hello AR example) would be very appreciated.

@wintersim https://codelabs.developers.google.com/?cat=Augmented+Reality

[zirman]

Here is a example app that loads a GLB file and renders it with Filament. All Sceneform dependencies are removed. This is not a framework. I hope this helps anyone that wants to get started. example

[nguyenbs]

@zirman : can you proceed your example with the shadows like the Filament iOS AR sample in https://github.com/google/filament/pull/1124? Or else can you give me any clue to start with? Many thanks

[zirman]

@nguyenbs I've updated my project to have shadows on detected horizontal planes.

[nguyenbs]

@zirman I managed to upgrade Filament to latest version 1.9.1 in the build.gradle of your sample but the app crashed when it start on my device (Samsung Galaxy S9). Can you help to update your example to Filament 1.9.1?

[zirman]

@nguyenbs I've updated the project to 1.9.1 but had to disable ambient lighting because it's causing Filament to crash.

[zirman]

Edit: @nguyenbs Fix is coming in 1.9.2 https://github.com/google/filament/issues/3119

[eschoenawa]

I'm also really interested in some consistent samples using ARCore and filament. I'd love to have similar samples to those that were provided for sceneform. Those samples were stellar and I could learn using sceneform just from them (and the occasional javadoc entry).

Sceneform provided access to AR to developers not that versed in computer graphics. I really hope that we one day get an API as simple as that one again, but for now, I want to learn how to do things like ViewRenderable in filament without having to comb through the open-sourced sceneform code.

[anahitaBarzegar]

@zirman Hello, I'm new in ARCore and OpenGl, I used your sample code with filament, but i have problem loading multiple object at the same time on the screen, please help me also I have another question about using filament vs OpenGl for ARCore android, which one is better to use (in terms of quality and its future changes)?

[atul161]

@zirman Hey ! Thanks for the example app. But it is not working as smooth as we wanted. Same thing i have rendered through unity and have a huge performance differnece. Seems like animation is laggy. Dont know the reason.