3D Image flips when moving my phone around in 360 view

[Sheeban-Wasi]

Hello, We have been working on a problem where we have a 3D image on web and we wish to move around in the image using our mobile device. We have used JS for getting the permissions from the user for the sensor everything works but at some angle the image flips upside down. The alpha beta and gamma changes dramatically.

I have attached the link with the post for better understanding.

Link 1 - https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multiply_test1.html (the angle combination is mentioned just below the link so that we can have better insight.

var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(a, b, c)

Link 2: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multiply_test2.html var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(a, c, b)

Link 3: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multi_tst3.html var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(b, a, c)

Link 4: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multiply_test4.html var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(b, c, a)

Link 5: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multiply_test5.html var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(c, a, b)

Link 6: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/multiply_test6.html var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(c, b, a)

All the links have different alpha beta and gamma placements in the .setFromEuler method. Please help me how can I solve the flip issue which you will realise when you will rotate the phone around.

Thanks

[andreasplesch]

Is there a way to test/reproduce without a phone ?

[Sheeban-Wasi]

Yes, if you open the link on the laptop then going to the console and there is an option below Sensors there is one more option orientation just switch it on and move the phone picture around. The image will rotate. Hope I made myself clear.

[andreasplesch]

Hm, on a desktop, Chrome devtools console does not seem to have a Sensors option, perhaps because the desktop does not have orientation sensors. I will try later on a laptop.

[Sheeban-Wasi]

Yeah may be !! but this is what it looks like on my laptop for reference.

[npolys]

https://www.sitepoint.com/how-to-simulate-mobile-devices-with-device-mode-in-chrome/

worked for my workstation

Emulated Mobile Sensors

These can be emulated in Chrome by choosing More tools, then Sensors from the Developer Tools main three-dot menu

On Thu, Feb 17, 2022 at 1:32 PM Sheeban @.***> wrote:

[image: Screen Shot 2022-02-17 at 1 30 25 PM] https://user-images.githubusercontent.com/83856744/154547291-d6f0de90-4737-4526-9bd5-ca744cfaf247.png

Yeah may be !! but this is what it looks like on my laptop for reference.

— Reply to this email directly, view it on GitHub https://github.com/x3dom/x3dom/issues/1195#issuecomment-1043280747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TSMZVGCI7UBUQCRVMPM3U3U5KFANCNFSM5OMMEECA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Nicholas F. Polys, Ph.D.

Director of Visual Computing Virginia Tech Research Computing

Affiliate Professor Virginia Tech Department of Computer Science https://people.cs.vt.edu/~npolys/ https://people.cs.vt.edu/~npolys/

[andreasplesch]

Found it:

[andreasplesch]

https://www.mathworks.com/help/robotics/ref/eul2quat.html may be helpful. Do you know what the expected order of the axes rotations are, for the values given by the phone ? I noticed that alpha tends to jump sometimes with tiny adjustments of the phone orientation by dragging the mouse.

[andreasplesch]

Just for reference: https://developer.mozilla.org/en-US/docs/Web/API/DeviceOrientationEvent

[andreasplesch]

https://w3c.github.io/deviceorientation/#deviceorientation

has info on the order of rotations.

[andreasplesch]

So it should be ZYX, eg. alpha, beta and then gamma. But the phone has Z up whereas X3D has Y up.

[andreasplesch]

https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles has more info. alpha is yaw, beta is pitch, gamma is roll, I think. So that seems to confirms that x3dom is using the ZYX order.

Wait:

x is alpha (yaw?) y is beta (pitch?) z is gamma (roll?)

But in the function formula:

x is r(oll) y is p(itch) z is y(aw)

So in x3dom's implementation gamma (z) is yaw, beta(y) is pitch and alpha(x) is roll.

So the function should be probably called with (e.gamma, e.beta, e.alpha) order.

But the output quaternion uses the phone coordinate system with Z up.

[andreasplesch]

Since the output quaternion assumes that Z is up, the output orientation needs to rotated so that positive Z points upwards. Since positive Z points inwards in X3D, it has to be rotated around X by 90 clockwise, I think.

[andreasplesch]

Also the vp orientation is the rotation from 0, 0, -1 to the desired orientation. Not sure what a zero rotation Euler angles from the phone mean. I think it means looking upwards. But no rotation in X3D means looking forward. So that would imply another 90 rotation around X, probably counterclockwise. Perhaps time for trial and error in devtools.

[andreasplesch]

In any case, the flipping is probably related to the use of TURNTABLE mode which tries to keep things vertical. Switching to EXAMINE mode seems to get rid of the flipping.

[andreasplesch]

You could just ignore the phone roll(gamma) to keep things vertical.

[Sheeban-Wasi]

Thank you so much for your detailed answers. I have tried few things as per the discussion.

For the vp I am setting the attribute like this:

V.setAttribute('orientation', r[0].z + ' ' + r[0].y + ' ' + r[0].x + ' ' + r[1])

[Sheeban-Wasi]

The viewpoint which I am updating is:

<viewpoint id='vp' centerOfRotation='0 0 0' description='"Plot Center 1"' fieldOfView='0.79' position='0 0 10'></viewpoint>

[Sheeban-Wasi]

For reference the script which we wrote is (//comments have some experimental code which I didn't removed maybe we can get an idea from there) - right now I can't see the flip but when I am keeping the phone in landscape mode or moving around it only moves vertically.

`function handleOrientation(event) { updateFieldIfNotNull('Orientation_a', event.alpha); updateFieldIfNotNull('Orientation_b', event.beta); updateFieldIfNotNull('Orientation_g', event.gamma);

incrementEventCount();

var V = document.getElementById('vp');
var degtorad = Math.PI / 180; // Degree-to-Radian conversion

// if the alpha beta gamma in degrees give them radians

var a = event.alpha * degtorad;
var b = event.beta * degtorad;
var c = event.gamma * degtorad;

// var q1 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0, 0, 1), event.gamma)
// var q2 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(1, 0, 0), b)
// var q3 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0, 1, 0), a)

// var q = q3.multiply(q2)

var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(a,b,c)

// var qr = q1.multiply(q2)
// qr = qr.multiply(q3);

// var a = q.getAxis()
var r = q1.toAxisAngle();
// r[1] = r[1].toFixed(2)

// console.log(r)
V.setAttribute('orientation', r[0].z + ' ' + r[0].y + ' ' + r[0].x + ' ' + r[1])
console.log('hello');

}`

[andreasplesch]

https://ntrs.nasa.gov/citations/19770024290 also has: which may match better the deviceorientation rotation sequence. The Wikipedia code has the X and Y sequence flipped. Closer scrutiny required.

[andreasplesch]

https://github.com/mrdoob/three.js/blob/02cf0df1cb4575d5842fef9c85bb5a89fe020d53/src/math/Quaternion.js#L206

has the ZXY case:

https://github.com/mrdoob/three.js/blob/02cf0df1cb4575d5842fef9c85bb5a89fe020d53/src/math/Quaternion.js#L206-L284

setFromEuler( euler, update ) {

        if ( ! ( euler && euler.isEuler ) ) {

            throw new Error( 'THREE.Quaternion: .setFromEuler() now expects an Euler rotation rather than a Vector3 and order.' );

        }

        const x = euler._x, y = euler._y, z = euler._z, order = euler._order;

        // http://www.mathworks.com/matlabcentral/fileexchange/
        //  20696-function-to-convert-between-dcm-euler-angles-quaternions-and-euler-vectors/
        //  content/SpinCalc.m

        const cos = Math.cos;
        const sin = Math.sin;

        const c1 = cos( x / 2 );
        const c2 = cos( y / 2 );
        const c3 = cos( z / 2 );

        const s1 = sin( x / 2 );
        const s2 = sin( y / 2 );
        const s3 = sin( z / 2 );

        switch ( order ) {

            case 'XYZ':
                this._x = s1 * c2 * c3 + c1 * s2 * s3;
                this._y = c1 * s2 * c3 - s1 * c2 * s3;
                this._z = c1 * c2 * s3 + s1 * s2 * c3;
                this._w = c1 * c2 * c3 - s1 * s2 * s3;
                break;

            case 'YXZ':
                this._x = s1 * c2 * c3 + c1 * s2 * s3;
                this._y = c1 * s2 * c3 - s1 * c2 * s3;
                this._z = c1 * c2 * s3 - s1 * s2 * c3;
                this._w = c1 * c2 * c3 + s1 * s2 * s3;
                break;

            case 'ZXY':
                this._x = s1 * c2 * c3 - c1 * s2 * s3;
                this._y = c1 * s2 * c3 + s1 * c2 * s3;
                this._z = c1 * c2 * s3 + s1 * s2 * c3;
                this._w = c1 * c2 * c3 - s1 * s2 * s3;
                break;

            case 'ZYX':
                this._x = s1 * c2 * c3 - c1 * s2 * s3;
                this._y = c1 * s2 * c3 + s1 * c2 * s3;
                this._z = c1 * c2 * s3 - s1 * s2 * c3;
                this._w = c1 * c2 * c3 + s1 * s2 * s3;
                break;

            case 'YZX':
                this._x = s1 * c2 * c3 + c1 * s2 * s3;
                this._y = c1 * s2 * c3 + s1 * c2 * s3;
                this._z = c1 * c2 * s3 - s1 * s2 * c3;
                this._w = c1 * c2 * c3 - s1 * s2 * s3;
                break;

            case 'XZY':
                this._x = s1 * c2 * c3 - c1 * s2 * s3;
                this._y = c1 * s2 * c3 - s1 * c2 * s3;
                this._z = c1 * c2 * s3 + s1 * s2 * c3;
                this._w = c1 * c2 * c3 + s1 * s2 * s3;
                break;

            default:
                console.warn( 'THREE.Quaternion: .setFromEuler() encountered an unknown order: ' + order );

        }

        if ( update !== false ) this._onChangeCallback();

        return this;

    }

[Sheeban-Wasi]

Since the output quaternion assumes that Z is up, the output orientation needs to rotated so that positive Z points upwards. Since positive Z points inwards in X3D, it has to be rotated around X by 90 clockwise, I think.

How to apply the extra rotation? Which constructor or method should I use to apply the extra rotation? I would really appreciate your help.

[Sheeban-Wasi]

Also the vp orientation is the rotation from 0, 0, -1 to the desired orientation. Not sure what a zero rotation Euler angles from the phone mean. I think it means looking upwards. But no rotation in X3D means looking forward. So that would imply another 90 rotation around X, probably counterclockwise. Perhaps time for trial and error in devtools.

var q1 = x3dom.fields.Quaternion.prototype.setFromEuler(a,b,c) var r = q1.toAxisAngle(); var newmat = x3dom.field.SFRotation(1,0,0,-1.578) var q2 = newmat.multiply(r) V.setAttribute('orientation', q2[0].x + ' ' + q2[0].y + ' ' + q2[0].z + ' ' + q2[1])

I tried the multiplication after q1.toAxisAngle to apply extra rotation. When should I do the multiplication? After the .toAxisAngle or before this in quaternion space?

Note: The SFRotation method or constructor doesn't work it's not correct. Which should I use?

[andreasplesch]

I think it may first make sense to implement the quaternion conversion for the ZXY case in your own function, to avoid potential confusion.

You were missing the new keyword in the SFRotation constructor. But you could probably just use Quaternion.parseAxisAngle. I think the order of application of rotations by quat. multiplication is secondRotation.multiply(firstRotation) (the same as in matrix multiplication).

Another approach is to wrap the viewpoint in three Transforms which then can apply the Euler angles in sequence. This may provide most control and confidence. It is weird that the deviceorientation api only provides Euler angles.

You can turn off Turntable mode in the NavigationInfo node.

[Sheeban-Wasi]

Thanks @andreasplesch for your responses. Finally we reached to a solution.

Link to the working solution: https://metagrid1.sv.vt.edu/~swmohd/npolys/ICAT2021/ICAT2021/Stroubles_2021/_Multi1_copy.html

`var runtime = null; var degreeToRadiansFactor = Math.PI / 180;

x3dom.fields.Quaternion.prototype.setFromEulerYXZ = function (alpha, beta, gamma) {

    var c1 = Math.cos( alpha / 2 );
    var c2 = Math.cos( beta / 2 );
    var c3 = Math.cos( gamma / 2 );
    var s1 = Math.sin( alpha / 2 );
    var s2 = Math.sin( beta / 2 );
    var s3 = Math.sin( gamma / 2 );

    this.x = s1 * c2 * c3 + c1 * s2 * s3;
    this.y = c1 * s2 * c3 - s1 * c2 * s3;
    this.z = c1 * c2 * s3 - s1 * s2 * c3;
    this.w = c1 * c2 * c3 + s1 * s2 * s3;
}

function deg2rad(deg){return deg * degreeToRadiansFactor;}`

function handleOrientation(event) {

var V = document.getElementById('vp');
var degtorad = Math.PI / 180;  // Degree-to-Radian conversion

var a = event.alpha * degtorad;
var b = event.beta * degtorad ;
var c = event.gamma * degtorad;

// phone rotation offset
// works now in LANDSCAPE MODE  and PORTRAIT MODE

if(!window.ondeviceorientation){
var q0 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0,0,1),-deg2rad(window.orientation));
 // var q0 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0,0,1),-Math.PI/2);
 var q = new x3dom.fields.Quaternion();

 q.setFromEulerYXZ(b, a, -c);  // our own implemented function

  var q1 = new x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(1,0,0),-Math.PI/2); 
// device orientation points upwards. rotate down to camera orientation

  q = q.multiply(q1);
  q = q.multiply(q0);
  var aa = q.toAxisAngle();

V.setAttribute("orientation", aa[0].x + " " + aa[0].y + " " + aa[0].z + " " + aa[1]);

}else{
 // var q0 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0,0,1),-Math.PI/2);
    var q0 = x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(0,0,1),-deg2rad(window.orientation));
    var q = new x3dom.fields.Quaternion();
    q.setFromEulerYXZ(b, a, -c);  // our own implemented function

    var q1 = new x3dom.fields.Quaternion.axisAngle(new x3dom.fields.SFVec3f(1,0,0), -Math.PI); // device orientation points upwards. rotate down to camera orientation

      q = q.multiply(q1);
      q = q.multiply(q0);
      var aa = q.toAxisAngle();

V.setAttribute("orientation", aa[0].x + " " + aa[0].y + " " + aa[0].z + " " + aa[1]);
}

[andreasplesch]

Super. That must have been some struggle. Hopefully, your solution can help others as well.