search

google-ar / arcore-android-sdk

ARCore SDK for Android Studio
https://developers.google.com/ar
Other
4.95k stars 1.22k forks source link

surfaces that configured to physical camera just got one frame after SetRepeatingRequest #1664

Open epiciskandar opened 3 weeks ago

epiciskandar commented 3 weeks ago

SPECIFIC ISSUE ENCOUNTERED

My purpose: use ARCore to get camera 3D position and posture data, and recording individual physical back camera, all at the same time.

When I tried using ARCore with Camera2 api which enables additional individual physical camera recording/preview, only the main(logical) camera got images that could be rendering, not those physical cameras.

If I call the Camera2 API doing the same thing, it works.

VERSIONS USED

STEPS TO REPRODUCE THE ISSUE

sorry for inconvenient providing full demo

  1. with GLSurfaceView, when onSurfaceCreated, create session with shared camera Session(this, setOf(Session.Feature.SHARED_CAMERA))
  2. configure shared session with auto focus, DepthMode.AUTOMATIC, and UpdateMode.LATEST_CAMERA_IMAGE
  3. call cameraManager.openCamera with wrapped callback
  4. in CameraDevice.StateCallback.onOpened:
    • call setCameraTextureName for rendering main video stream openGL rendering, and configure a list of List<OutputConfiguration> with surfaces from sharedCamera.arCoreSurfaces, alongside with surfaces that comes from jetpack compose UI constructing which configured with specified physical camera by calling OutputConfiguration.setPhysicalCameraId
    • store surfaces from list above, for further repeating capture request
    • cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD)
    • SessionConfiguration(SessionConfiguration.SESSION_REGULAR, outputConfigs, ...) with wrapped StateCallback
  5. in CameraCaptureSession.StateCallback.onConfigured, call session.setRepeatingRequest with surfaces saved previously, it is sure that createCaptureRequest and setRepeatingRequest have the same target/surface config.

Now, the logical camera render to GLSurfaceView corretly, but SurfaceViews that should showing physical camera preview just got one frame.

video below:

https://github.com/user-attachments/assets/8631bd04-f251-4768-ba43-7853f02e1228

in the video, the beginning half shows calling ARCore and Camera2, notice that sub previews only got updated one frame. at the end, video shows how the preview should be shown to views(and currently main preview have bug, not related here) if only using Camera2.

WORKAROUNDS (IF ANY)

ADDITIONAL COMMENTS

The ARCore update is driven by sharedSession.update() at GLSurfaceView.Renderer.onDrawFrame, and I got camera position data from frame.camera.pose

epiciskandar commented 2 weeks ago

This problem could be fixed by "double commit" like this:


val callback = object : CameraCaptureSession.StateCallback() {
    override fun onConfigured(session: CameraCaptureSession) {
        this@MainActivity.session = session
        val rr = object: Runnable {
            override fun run() {
                session.setRepeatingRequest(
                    requestBuilder.build(),
                    cameraCaptureCallback,
                    cameraHandler
                )
            }
        }
        cameraHandler.postDelayed(rr, 0)
        cameraHandler.postDelayed(rr, 100)  // could not be too small, and must post two times
    }
}

but I don't know why

edit: This only works on Samsung, on the Pixel 7, double commit will stop main(logical) camera receiving data, and got this error log:

2024-08-23 16:33:44.464  7194-7376 gesture         com.gesture             E  [SurfaceTexture-0-7194-0] detachFromContext: SurfaceTexture is not attached to a GL context
2024-08-23 16:33:44.464  7194-7376  ArCore-TextureStore     com.gesture             E  attachTexImage: calling detachFromGLContext failed:Error during detachFromGLContext (see logcat for details)

But add delay time will make it works on Pixel 7 too, like this:

        cameraHandler.postDelayed(rr, 100)
        cameraHandler.postDelayed(rr, 300)