Textures and device pose at 60Hz

[TrackBallPower]

Not sure if this has been requested already, but i thought i'd give it a go. 60fps capable mobile phone camera's have been around for quite some time now. Samsung, for example, has been bringing out 60fps capable camera's since the S6.

I noticed that ARCore currently supports Textures and device pose at only 30Hz. This is starting to be become a limiting factor in some developments, such as the implementation of positional tracking (6DOF) in Quake Gear VR. The camera tracking is simply not fast enough. See also: https://www.reddit.com/r/GearVR/comments/9lzqpv/quake_gear_vr_samsung_s8_decoupled_aiming_and

Are there any plans to implement Textures and device pose at 60Hz?

[inio]

Thanks for the suggestion. This is something we're aware of and would like to support, but there's a lot of things we'd like to support and only so much time in the quater.

We've experimented with running 60FPS but it introduces problems with increased device heating, decreasing battery life, and disappointing performance in lower light situations.

[TrackBallPower]

Thanks for your feedback inio. If that's ok with you, let me try to break down those reasons:

1. increased device heating
   • Lots of new mobile devices have heat pipe technology to mitigate overheating
   • Lots of mobile devices will overheat from many high performance games anyway

My guess is that many higher performance mobile devices will be able to handle 60hz without overheating, so why not make the 60hz an optional setting?

2. Decreasing battery life That can be easily mitigated by plugging in the charging cable, or simply accepting that the device will consume more battery. Again, making the 60hz an optional setting can also provide a solution in this case.

3. Disappointing performance in lower light situations The obvious answer is that it can be mitigated by switching on the lights. However, lots of newer mobile phones can handle low light environments quite well. Those people will also benefit from a 60hz optional setting.

[wisockijunior]

thats it, we really need it, for inside out positional tracking, could it be integrated in Daydream for 6DoF on compatible phones?

[inio]

Unfortunately, ARCore is just not well suited for providing 6DOF VR tracking. While ARCore and WorldSense are powered by the same core algorithms, the configuration used for ARCore is optimized for handheld AR use and will result an extremely poor VR experience.

[kolboch]

@TrackBallPower yeah just accept it... IMO whenever there is something higher in quality devs would use that as default, then it works negatively for opinion of AR in general as people would tend to think it has to eat their battery and overheat cpu.

[TrackBallPower]

Well the question is then if 60hz would benefit handheld AR as well. IMHO, it is never good if software is unable to make use of the capabilities of better performing hardware. It gives me the impression that scaling in ARCore needs more attention.

[ChristophGeske]

There is a work around to this issue. When you recive the new position data you only move the camera half the way and you use the rest of the way to fill the missing frame. This introduces a bit of lag but I did not notice it in VR. I also heard rumors that the Pixel phones record in 60fps can someone confirm this. If you like to try the described work around you can check out my positional tracking project using daydream and gear vr. https://github.com/ChristophGeske/ARCoreInsideOutTrackingGearVr

[TrackBallPower]

Hi ChristophGeske, coincidentally i already advised the developer of Quake Gear VR to use your workaround, because he couldn't get a prediction algorithm to function properly:

https://www.reddit.com/r/GearVR/comments/9lzqpv/quake_gear_vr_samsung_s8_decoupled_aiming_and/e7r5kno/

I didn't get a response from him, however Quake for Gear VR with 6DOF/ARCore has been released just a few hours ago:

https://www.reddit.com/r/GearVR/comments/9p3icr/released_3dof_controls_and_positional_tracking/

If you're interested, maybe you could have a go with it and give him some suggestions for improvement?

[andrelevi]

While 30hz poses can be interpolated, I think 60hz camera feed updates is a must-have in order to match ARKit. ARKit on an old iPhone 6s works fine with 60hz camera feed and poses.

[inio]

ARCore's use of the android motion sensors doesn't prevent your app from using them as well. You should be able to forward integrate off of the last ARCore pose using that data. ARCore is actually doing this already: our visual-inertial-odometry only updates around 10 times per second and the live pose is forward-integrated based on the IMU readings.

Note: for computing velocity, create an anchor at the camera each frame and measure relative pose on the next frame. Without this you'll get velocity spikes when we update our IMU-based tracking from the slower VIO data, or the VIO from SLAM data.

[wisockijunior]

ARCore's use of the android motion sensors doesn't prevent your app from using them as well. You should be able to forward integrate off of the last ARCore pose using that data. ARCore is actually doing this already: our visual-inertial-odometry only updates around 10 times per second and the live pose is forward-integrated based on the IMU readings.

Note: for computing velocity, create an anchor at the camera each frame and measure relative pose on the next frame. Without this you'll get velocity spikes when we update our IMU-based tracking from the slower VIO data, or the VIO from SLAM data.

Can we at least use Qualcomm XR embedded solutions, like on Snapdragon 835 and Snapdragon 845? They already provide SLAM at 60Hz without any processing penalties, as the XR1 platform also has special optimizations for Augmented Reality (AR) experiences with Artificial Intelligence (AI) capabilities offering better interactivity, power consumption and thermal efficiency. The platform also features advanced vision processing capabilities fundamental for technologies like Visual Inertial Odometry (VIO), which lets users move around in the virtual world or interact with augmented objects in an AR experience. An integrated sensor hub and optimized sensor fusion abilities allows users to experience rich interactions with motion to photon latencies well below the scientifically required 20ms.

[b29b]

it doesnt seem fair that google added 60fps to the pixel devices and left everyone else does any 2020 device other than pixel work at 60fps?

[Piipperi]

Is ARCore still as of 2023 stuck at 30 FPS on any device other than Pixel? I'm trying to create a Unity game, which runs buttery smooth 60 FPS on iOS, but on Android flagship like Galaxy S22, there's a lot of latency and no way to set it to go above 30 FPS. Google doesn't sell Pixel's in my region neither, so I have no good way to test how well does my game work on Android...

[raju535482]

They are not taking this seriously, instead of solving many bugs. and stablizing the existing SDK. They just keep adding new stuff and make it more worse.

[xiaofengzhang777]

• @inio I kindly request your assistance in resolving this issue I am encountering. If you could take the time to look into this matter, I would greatly appreciate it:
  • https://github.com/google-ar/arcore-android-sdk/issues/1600