ArSession_update fails after pause/resume, even with session re-creation

[robbiesri]

SPECIFIC ISSUE ENCOUNTERED

We've been getting reports about UE4 hitting failures after pause/resume with ARCore. After pause/resuming 2/3 times, I can see that we can't get an updated camera feed.

After initial debugging of the issue, I found that ArSession_update was returning AR_ERROR_FATAL (ArSession_update docs). At this point, my assumption is that the session is hosed, and the only thing I can do is tear down this session, and create a new session.

I can successfully destroy the old session, and create a new sesson. I chug along until I get back to ArSession_update and...it fails in the exact same way.

I tried to recreate the issue in the SDK samples. I couldn't get the session to fail with pause/resume.

VERSIONS USED

• Android Studio: 4.1.2
• ARCore SDK for Android: 1.22.0
• Device manufacturer, model, and O/S: Google Pixel 4 XL, Android 11
• Google Play Services for AR (ARCore): versionName=1.27.212430123
• Output of adb shell getprop ro.build.fingerprint: google/coral/coral:11/RQ2A.210405.005/7181113:user/release-keys

STEPS TO REPRODUCE THE ISSUE

1. Create sample project from UE4 handheld AR template (reproduces on UE 4.27, reported on UE 4.24+ with different ARCore SDKs + Android versions)
2. Package and launch the app on a ARCore device
3. Switch between the deployed app and another app a couple times. Eventually, the camera will stop working.

ADDITIONAL COMMENTS

• NDK: r21e aka 21.4.7075529
• Let me know if you need me to package up a sample. Wasn't sure if you already have UE4 build workflows on your side.
• I don't 'mind' the failure. I just don't know what I'm supposed to do about this failure. Session re-creation feels like a big hammer, and I'd expect that to work.
• I briefly chatted with Phoenix about this, and he asked a good question: does ArSession_resume succeed before ArSession_update fails? ArSession_resume is indeed successful, which makes the update failure even more confusing. I'd expect resume to fail 😢
• I do see some spam in logcat about the session NOT_YET_AVAILABLE, but Phoenix told me this is expected: | 10-12 | 15:21:30.646 | 20330 | 20464 | E | native | at third_party/arcore/ar/core/session.cc:2721 [type.googleapis.com/util.ErrorSpacePayload='ArStatusErrorSpace::AR_ERROR_NOT_YET_AVAILABLE'] |

[devbridie]

Hi Rob, would be great if you could package a sample; please send it to dereckbridie@google.com. If the UE4 AR template is OSS, could you tell me where I could find that as well?

[robbiesri]

Thanks for the quick response, Dereck!

Here is the template soure: https://github.com/EpicGames/UnrealEngine/tree/4.27/Templates/TP_AEC_HandheldARBP But you'd just sync the repo, build the engine, and then create a new project based on the Handheld AR template.

I'll package up a debug and development build to send to you right meow

[devbridie]

Repro'd in the sample you sent; that's strange. It seems that the session update is being called on a thread that isn't EGL compatible. Will try to look into this, but I'm not an expert on this platform.

[robbiesri]

Hey Dereck, thanks for the word. I think that makes sense, since we call update on the game thread, which typically doesn't have access to the GL context. That being said, I'm surprised that it hasn't fallen over previously. I don't see an explicit requirement that the GL context is accessible in a particular thread, but I do see that ArSession_update can return an error about a missing GL context (AR_ERROR_MISSING_GL_CONTEXT), which implies the GL context is accessible on the thread that calls ArSession_update.

Edit: actually, I need to be more sure about which threads do or do not have access to the GL context. I could be incorrect my assumption.

[CanYouEatIt]

Was there any fix for this?