DeviceOrientationControls iOS initial heading/alpha correction

[spoxies]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

On iOS using DeviceOrientationControls, the 'north' is set to whatever direction you happen face when initialising the controls. This issue belongs to a PR https://github.com/AR-js-org/AR.js/pull/467 (maybe first merge https://github.com/AR-js-org/AR.js/pull/464).

If the current behavior is a bug, please provide the steps to reproduce.

Get a iOS device and run the AR-js-org sample code (examples/location-based). You will find that the north (alpha) will be set to 0 whatever direction you are facing at that time.

https://github.com/WebKit/webkit/blob/main/Source/WebCore/platform/ios/WebCoreMotionManager.mm

Please mention other relevant information such as the browser version, Operating System and Device Name

iOS 13 until iOS 16 (tested). Luckily a lot of the work had been done https://github.com/mrdoob/three.js/pull/4577/commits/34c681013eb4403860c5a0e6fa29a91f497b6117, https://github.com/MasterJames/three.js/commit/208b9753777f81692a2ecedfa74f520b8299981a. But that was later removed/cleaned/overlooked in the original Three.js https://github.com/mrdoob/three.js/pull/13000. But it is still relevant and a minor fix.

What is the expected behavior?

The scene should match and be facing, the correct direction.

If this is a feature request, what is motivation or use case for changing the behavior?

Add support to WebKit/iOS for DeviceOrientationControls

[sahilimmco]

Hi I would like to know how the changes made by you to solve alpha/header correction in three.js based AR to be implemented in A-frame location based ar.js

[spoxies]

I just added the same fix for aframe in the PR if you look for that line in you build and replace it: https://github.com/AR-js-org/AR.js/pull/467/commits/fed4b7160c423dc57b6690f19a54ffaa358621d0

[sahilimmco]

Hi Thanks for your reply. unfortunately I couldn't found that line on my build. I am currently using "https://github.com/AR-js-org/AR.js/blob/3.4.0/aframe/build/aframe-ar.js" this build for developing application. would you please help me to integrate that fix in to this build.

I tried to update the code after updating the code, objects is sticking on the camera. The orientation feature itself is not working

[spoxies]

@sahilimmco

You are probably looking for

if(A){var B=A.alpha?I.Math.degToRad(A.alpha)+g.alphaOffset:0,

and replace it with something like

if(A){var heading=A.webkitCompassHeading || A.compassHeading;var B = A.alpha || heading ? I.Math.degToRad(heading ? 360 - heading: A.alpha || 0) + g.alphaOffset: 0,

I'm not sure if the above works for you but please post following up questions elsewhere (e.g. stackoverflow).

[kalwalt]

Hi @spoxies, thank you for this, does this issue is related only to iOS devices? Have you tested with an Android device?

[spoxies]

Hi @kalwalt, Yes this only relates to iOS devices.

It is tested on Android in the sense that webkitCompassHeading and compassHeading properties are not present. Hence const heading will be undefined and the existing calculation is unchanged and does not error.

Expand - Detailed answer

```javascript const heading = device.webkitCompassHeading || device.compassHeading; // undefined (on Android) ``` Hence the previous calculation as before will be executed on Android: **Previous calculation:** ```javascript const alpha = device.alpha // if(device.alpha) => #[IF~1] ? MathUtils.degToRad(device.alpha) + scope.alphaOffset // #[IF~1]==true : 0; // #[IF~1]==false ``` **Added/combined calculation:** ```javascript const alpha = device.alpha || heading // if(device.alpha || heading) => #[IF~1] ? MathUtils.degToRad( heading // if(heading) => #[IF~2] ? 360 - heading // #[IF~2]==true : device.alpha || 0) // #[IF~2]==false (|| 0 is redundant ) + scope.alphaOffset : 0; // #[IF~1]==false ``` I think it is very unlikely that `compassHeading` will become part of the spec (and also adopted by Android/Chromium), but if it does I think it can be assumed that the parameter type be adopted also. The most likely scenario is that it will remain a very [iOS/Webkit specific](https://lists.w3.org/Archives/Public/public-geolocation/2011Jul/0014.html) bypass and it might disappear but it is there for 11 years all-ready I think they are commited. Could also be: ```javascript const heading = device.webkitCompassHeading || device.compassHeading || false; // false (on Android) ```

[sahilimmco]

Thanks for your reply .Now the code is working for me but my problem is when I change my device from portrait to landscape the position of the object is shifting in iOS but while checking in android device it is working fine ,there is no difference in the position of object

[spoxies]

@sahilimmco Thanks for reporting, you are right and I tried some other things but they turn out to be unpredictableas well.
I'm not sure if I will be able to fix this but I'll give it a shot.

[sahilimmco]

@spoxies I really appreciate your efforts. If you have any idea about what causing the problem, Please do share with me. I will do a parallel try along with you.

[spoxies]

This needs to be validated:

If I have interpreted correctly (by looking at the calculations of which I can gasp 60%), I assume at this point, that you can calculate a compassheading from alpha, beta, gamma, but you can not calculate a alpha from a heading as that depends on the combination of the other axises and the calculation differs based on the position/values of those axis.

And if that is true, then IRL the starting position of the device (holding upright , flat on table in landscape) and thus the resulting calculation order of the compassheading determines if the heading can be used as an alpha offset.

Thus 'm unable to reliably reconstruct the correct alpha as basically the whole calculation rotates. I can not exclude my own inability, however at the bottom of my deep dive I found more people stranded and issues closed out of desperation. But as I can not fully gasp the involved calculations , I can not 100% state that the above is true, so it needs to be validated by someone who is more Maths capable.

When

I momentarily have exhausted my capabilities. I know a whole lot more, but none lead to fixing this issue within javascript it self. With the research done I have been able to quickly write a cordova plugin. But this is a headsup that a fix might not be coming soon in this repo from my side.

---

Additional info

### Additional #### Can do (?) Until this point I have been persistent in trying because for example w3c simply said (in 2008) that you can do a "[World-based calibration on iOS](https://www.w3.org/2008/geolocation/wiki/images/e/e0/Device_Orientation_%27alpha%27_Calibration-_Implementation_Status_and_Challenges.pdf)" using compassHeading. And also [the person](https://github.com/richtr) who has a lot to do with the standards and on top of that also contributed to THREE.js devicecontrols and Full Tilt deviceOrientation, says it [can be done](https://dev.opera.com/articles/w3c-device-orientation-usage/#:~:text=iOS%2Dbased%20browsers%20currently%20return%20deviceOrientationData.alpha) that way. On top of the assumptions above I have tried [combining](https://github.com/spoxies/AR.js/commit/f8a41bfe652bb14c4f977eaea35d738c015b0208#diff-990642202acef1b5670b08c9d85498648f9f67242d3a8d63ed6faf8fbe78ac12) multiple fixes and combinations (for example this [commit](https://github.com/MasterJames/three.js/commit/9447ec6b30c09df5bfb9214f8946b245b216deaf#diff-85275d8f73a5125c43083463a8ba301c6cba12264ae82253c3c898d53f781eca) mixed with the non MIT [Full-Tilt](https://github.com/adtile/Full-Tilt/blob/master/src/DeviceOrientation.js). But it gives me the correct result in 50% of the cases. And I assume this is because of reason given in the first lines of this comment. Further more i found that the `screenorientation` event will be `0` if the screen rotation is locked at the app or device level. However the WebKit calculations do not alway take this into account. #### W3C Standard Although conclusion in the w3c discussion seems to be that the `deviceorientationabsolute` event should be implemented to solve the [discrepancies](https://github.com/w3c/deviceorientation/issues/6) and give more [reliable/usable result](https://github.com/w3c/deviceorientation/issues/21), it seems that at this moment there is nothing on the horizon in this direction for WebKit because as far as they are concerned the standard [is supported](https://webkit.org/status/#?search=deviceorientation) . #### Now what What I also have done is overturned the Webkit and Chrome iOS source code to try and understand what is going on. This gave me enough of an understanding to throw together a cordova plugin that calculates and passes the correct `alpha, beta, gamma` and that works reliable. In the coming period I will try to rebuild that plugin so it can be contributed to the continued [cordova-plugin-device-motion ](https://github.com/apache/cordova-plugin-device-motion ) to at-least polyfill the missing `deviceorientationabsolute` in that environment.

[kalwalt]

@spoxies thank you for your effort!Unfortunately i can't help you much with, i haven't a device to test and no experience on iOS side.

[giangm9]

You can storage the initial compass value in somewhere

if (IS_IOS) {
   if (scope.compass == null && device.webkitCompassHeading) {
       scope.compass = MathUtils.degToRad(device.webkitCompassHeading);
   }
}

Then rotate the camera after the setObjectQuaternion

setObjectQuaternion(
  scope.object.quaternion,
  alpha,
  this.smoothingFactor < 1 ? beta - Math.PI : beta,
  this.smoothingFactor < 1 ? gamma - this.HALF_PI : gamma,
  scope.compass,
  orient
);

if (IS_IOS)
  scope.object.rotation.y += (Math.PI * 2 - scope.compass);

[json91-dev]

I solved IOS issue by below steps.If you can change DeviceOrientationControls, it's worth a try.

1. Get deviceOrientation event and calculate quaternion setObjectQuaternion function.
2. Export euler angles x,y,z from calculated quaternion.
3. Add (360-webkitCompassHeading) to euler y.
4. Calculate final quaternion from calculated euler
5. Apply final quaternion to camera

/** Calculate the rotation value of the registered target (camera) in real-time through quaternion calculation **/
this.update = function ({ theta = 0 } = { theta: 0 }) {
  if (scope.enabled === false) return
  const device = scope.deviceOrientation
  if (device) {
    const alpha = device.alpha ? THREE.MathUtils.degToRad(device.alpha) : 0 // Z
    const beta = device.beta ? THREE.MathUtils.degToRad(device.beta) : 0 // X'
    const gamma = device.gamma ? THREE.MathUtils.degToRad(device.gamma) : 0 // Y''
    const orient = scope.screenOrientation ? THREE.MathUtils.degToRad(scope.screenOrientation) : 0 // O

    if (isIOS) {
      // Calculate the quaternion first through deviceOrientation
      const currentQuaternion = new THREE.Quaternion()
      setObjectQuaternion(currentQuaternion, alpha, beta, gamma, orient)

      // Extract the Euler angles from the quaternion and add the heading angle to the Y-axis rotation of the Euler angles
      const currentEuler = new THREE.Euler().setFromQuaternion(currentQuaternion, 'YXZ')
      console.log(currentEuler.x, currentEuler.y, currentEuler.z)

      // Replace the current alpha value of the Euler angles and reset the quaternion
      currentEuler.y = THREE.MathUtils.degToRad(360 - device.webkitCompassHeading)
      currentQuaternion.setFromEuler(currentEuler)
      scope.object.quaternion.copy(currentQuaternion)
    } else {
      // Directly calculate through the deviceOrientationAbsolute event (Android)
      setObjectQuaternion(scope.object.quaternion, alpha + theta, beta, gamma, orient)
    }
  }
}

In this way, you can correctly find the north direction initially on iOS as well. DeviceOrientaion.jsx.zip

[nickw1]

Can someone else with an iOS device verify that @json91-dev's fix works on iOS? If so I will merge and check it's still working on Android.

Thanks.