Face center and region pose error increases significantly with distance from the camera field centre. For regions this is immediately obvious, for example the nose in the augmented face demo is pushed out to the sides in landscape mode (see images attached). Objects based on the face mesh and center pose register well across the field however (see images). The region and center pose orientations remain fixed relative to each other, however the mesh is changing in local coordinates (See the remaining images - the mesh is rendered with a fixed pose so the rotation of the mesh can be clearly seen for the off centre images). The mesh change in combination with the center pose produces the global registration, not the centre pose alone.
VERSIONS USED
Android Studio: 2020.3.1
ARCore SDK for Android: 1.43
Device manufacturer, model, and O/S: Motorola G6 (same results for arcore on IPhone 8)
Google Play Services for AR (ARCore): versionName=1.43.240930403
Output of adb shell getprop ro.build.fingerprint: motorola/ali_n/ali_n:8.0.0/OPS27.82-87/108:user/release-keys
STEPS TO REPRODUCE THE ISSUE
The errors are always present, and do not require special conditions.
The images are produced from the augmented faces demo, modified to render the canonical_face_texture.png from the sdk, or the nose object. The fixed orientation pose is generated using float[] modelMatrix = { 0,-1f,0,0, 1f,0,0,0, 0,0,1f,0, 0,0,-0.5f,1f };
WORKAROUNDS (IF ANY)
Apply heuristic rotation offsets as a function of face position relative to camera.
Calculate a local pose based on eye and nose mesh vertices, then apply current face pose to get more accurate face pose for region poses and tracking.
ADDITIONAL COMMENTS
The error is frustrating for accurate render of region objects, and also for accurate head tracking with translation, which has good uses besides graphic rendering.
It appears that since face registration is achieved very well, the gross change in the face mesh coordinates could be shifted to the center pose (and to correct the region poses), so that a much more accurate physical head pose is given for non central positions.
Face center and region pose error increases significantly with distance from the camera field centre. For regions this is immediately obvious, for example the nose in the augmented face demo is pushed out to the sides in landscape mode (see images attached). Objects based on the face mesh and center pose register well across the field however (see images). The region and center pose orientations remain fixed relative to each other, however the mesh is changing in local coordinates (See the remaining images - the mesh is rendered with a fixed pose so the rotation of the mesh can be clearly seen for the off centre images). The mesh change in combination with the center pose produces the global registration, not the centre pose alone.
VERSIONS USED
adb shell getprop ro.build.fingerprint: motorola/ali_n/ali_n:8.0.0/OPS27.82-87/108:user/release-keysSTEPS TO REPRODUCE THE ISSUE
The errors are always present, and do not require special conditions.
The images are produced from the augmented faces demo, modified to render the canonical_face_texture.png from the sdk, or the nose object. The fixed orientation pose is generated using float[] modelMatrix = { 0,-1f,0,0, 1f,0,0,0, 0,0,1f,0, 0,0,-0.5f,1f };
WORKAROUNDS (IF ANY)
Apply heuristic rotation offsets as a function of face position relative to camera.
Calculate a local pose based on eye and nose mesh vertices, then apply current face pose to get more accurate face pose for region poses and tracking.
ADDITIONAL COMMENTS
The error is frustrating for accurate render of region objects, and also for accurate head tracking with translation, which has good uses besides graphic rendering.
It appears that since face registration is achieved very well, the gross change in the face mesh coordinates could be shifted to the center pose (and to correct the region poses), so that a much more accurate physical head pose is given for non central positions.