ExternalTexture for custom renderer

[jpsarda]

I read this issue here about making a custom renderer for playing video ; https://github.com/google-ar/sceneform-android-sdk/issues/169

Now ExternalTexture is out and google has released a code sample playing a video through it : https://github.com/google-ar/sceneform-android-sdk/blob/master/samples/chromakeyvideo/app/src/main/java/com/google/ar/sceneform/samples/chromakeyvideo/ChromaKeyVideoActivity.java

My problem with this is that I don't just want to render a video, I have a complex view mixing images, and videos, and other things (like a small openGL game). So I need to render this view on a sceneform node. Could it be possible to render that view into an ExternalTexture ? The only problem would be that, contrary to ViewRenderable, the user interactions with my android view would not be managed.

But is it theoretically possible ? Do you see another way to achieve what I'm looking for ? If rendering an android view to an ExternalTexture is possible, how would you manage the user interactions with that view ? I was thinking about adding a ViewRenderable view at exact same anchor and same size/orientation, and passing the touch events to the android view, but is this possible ?

[romainguy]

That is exactly what ViewRenderable does, it renders the View into an external texture. Is there a reason you cannot use ViewRenderable directly? What type of View are you using exactly?

[jpsarda]

Thx very much for the reply. The view I would like to render is a kind of rich media/game engine. It contains in its hierarchy several ImageViews and several TextureViews rendering videos through MediaPlayer. All views are mixed, an ImageView can hide a piece of a TextureView, and reversally.

As far as I can know, the TextureViews are not supported by ViewRenderable? I'm sorry I can't test it right now because my phone is not supported by ARCore, that will be fixed soon.

My initial question was, considering that TextureViews are not supported by ViewRenderable, would it be possible to render a whole android view that contains ImageViews and TextureViews into an ExternalTexture ? But the answer is probably "yes but TextureViews won't be rendered" because you just said "That is exactly what ViewRenderable does" .

Since then I tried a few more things. To solve the problem of TextureViews not being supported by ViewRenderable, I've been working on a CustomVideoView. It's a ViewGroup which contains in its hierarchy an ImageView and a TextureView (hidden, out of bounds). The TextureView renders the video with MediaPlayer, and 30 times per seconds I use the TextureView.getBitmap method to copy its content into the ImageView. That looks scary and dirty, but it works and the performance is better than I expected. But this is dirty, if you see a better way to do this, let me know. Anyway, I can't test it with ARCore right now, but for me this has good chances to work fine.

The other possibility I had was to re-write this rich media/game engine using ARCore ExternalTexture for playing videos and displaying images. But that's a very painful path for dev and maintenance.

Let me know it's not clear.

Also, one last question. I plan to adapt a couple of small OpenGL games (GLSurfaceViews) to ARCore. I just want to show the games as they are through a ViewRenderable in 2D plane anchored in the 3D AR space. Once again, GLSurfaceViews are not supported, so my plan was first to make the games use TextureViews instead of GLSurfaceViews (which seems not to be very difficult to do), and use the same hack I dead for my CustomVideoView. Let me know if you see better alternatives.

[romainguy]

You are right that TextureView and SurfaceView are not supported by ViewRenderable. TextureView could be made to work in theory though, but that would require changes in Sceneform.

You don't need to change your games to use TextureView. Whether you are using a GLSurfaceView or a TextureView you are rendering into a Surface in the end. All you need is the ability to set the rendering Surface in the game. You can check the UI helper we built for Filament (Sceneform's renderer) that makes it easy to use either a SurfaceView or a TextureView.

[jpsarda]

Support of TextureView would be so great indeed.

OK I get the idea for the gl views, the idea would be to render the gl directly to the ExternalTexture. I'll look into the UI Helper class. Thank you very much.