2D "Steering" for flight simulators

[CasperH2O]

For space sims/flight games a 2D steering using yaw and pitch. Will require sensor fusion/AHRS to handle the rotation around the Z axis for which we have no acceleration value ie prevent drift. Additionally it might be needed to use separate X and Y multipliers to handle wide devices.

[Zebrazilla]

If I'm understanding this correctly it would potentially be useful for panning in a top-down game, like RTS or old school CRPGs.

[CasperH2O]

@Zebrazilla , not sure what you mean? Someone mentioned he would try Star Wars Squadrons with HC, but then I realized he would need to be able to pitch his device up and down and while held there his spaceship would go up / down. Basically like a joystick. This is different from the camera aiming stuff as that "stops" rotating the moment you stop rotating the device. Where as with steering, you want the joystick to be held in a certain position relative to the angle the device is being held.

[Zebrazilla]

@CasperH2O Please correct me if I'm thinking of this incorrectly; steering in the sense that it is already implemented makes it so tilting yaw axis applies a "constant" value, depending on how far you tilt, as long as that tilt is held. Returning to no tilt stops the value applied and doesn't apply a value in the opposite direction in doing so. You can then map that so any value on left and right joystick results in keyboard left and right, or a and d. What you are suggesting would allow for pitch to map to up and down similarly, correct? I am then thinking it would be possible to play a top-down game where you navigate with arrow keys or wasd around the map. Holding still with no tilt holds the camera still, tilting in yaw and/or pitch would then allow to pan in a direction, and keep panning until the user "levels out" again and holds the device so that's effectively no yaw or pitch. Again I may be thinking of this wrong, I am quite new to this gyro stuff after all 😅

I tried the steering option in a top-down game once before and I then managed to make it so yaw would pan the camera quite well, but then it only worked left and right. Default option (is it player space?) worked in all directions but made it so tilt did pan the camera around the map, but leveling out so that there was no tilt more or less returned to the same location as then the opposite movement was applied.

[CasperH2O]

@CasperH2O Please correct me if I'm thinking of this incorrectly; steering in the sense that it is already implemented makes it so tilting yaw axis applies a "constant" value, depending on how far you tilt, as long as that tilt is held. Returning to no tilt stops the value applied and doesn't apply a value in the opposite direction in doing so.

The currently implemented steering feature, will, depending on the device angle around the Roll axis set the selected joystick output. Say, if you have configured a max range of 30 degrees, tilting the device to -30 will set the joystick X value to -1, tilting the device to 15 degrees will set the joystick X value to 0.5.

You can try this out yourself on a site like: https://gamepad-tester.com/ be sure to create the settings in the default profile or make one specific for your browser.

You can then map that so any value on left and right joystick results in keyboard left and right, or a and d. What you are suggesting would allow for pitch to map to up and down similarly, correct?

Right now, Handheld Companion does not allow for mapping to keyboard left right and A and D, only joystick left or right.

I am then thinking it would be possible to play a top-down game where you navigate with arrow keys or wasd around the map. Holding still with no tilt holds the camera still, tilting in yaw and/or pitch would then allow to pan in a direction, and keep panning until the user "levels out" again and holds the device so that's effectively no yaw or pitch. Again I may be thinking of this wrong, I am quite new to this gyro stuff after all 😅 I tried the steering option in a top-down game once before and I then managed to make it so yaw would pan the camera quite well, but then it only worked left and right.

If we're talking classic RTS games like Age of Empires or Red Alert we would have to find ways to control both the view (keyboard arrows or WASD) and the mouse. Handheld Companion is not yet that sophisticated, but it's definitely interesting. If you would like to explore more in this area you can connect HC's emulated DS4 controller to an application called JoyShockMapper. It's very advanced and has tons of configurability. I think you are referring to something like this:

https://youtu.be/7_wBa3Taefc?t=162

And

https://youtu.be/GDqfCjGhodk?t=51

Default option (is it player space?) worked in all directions but made it so tilt did pan the camera around the map, but leveling out so that there was no tilt more or less returned to the same location as then the opposite movement was applied.

Player space is an invention by Jibbsmart mostly meant for aiming with weapons in first and third person games, it's a step up from using purely gyrometer data with a multiplication factor, giving a very grounded feeling while also providing the option to look around left and right with your device.

There's a whole article on it here: http://gyrowiki.jibbsmart.com/blog:player-space-gyro-and-alternatives-explained

The regular racing steering would be done by rotating the device around the roll axis, where as a space sim/flight game would "steer" by rotating the device around the pitch and yaw angles. To be prototyped and evaluate...

If you'de like to talk more we can be found on the Aya Neo Discord, sub channel Controller Service.

[Yoo-O]

@CasperH2O I want to have a try to add pitch based on the current steering mode. The output is a little shaking and I plan to use some filters like kalman to solve it. And now I have some problems, can you offer some help? Thanks very much.

1. The initial position of GRYO is horizontal, this is not a problem for steering in racing games, but not very suitable for the pitch direction. And I want to change the initial position to the position when GYRO toggle button is pushed/released(decided by the GYRO enable mode ), where is the code function can get the position info when GYRO toggle button is pushed/released?
2. What is the angle unit of AngleToJoystickPos function, degree or radian?

[CasperH2O]

@Yoo-O sounds great! I've been thinking about this on and off myself.

1. If I understand correctly you want to use the position angle change in a relative manner, instead of the absolute position of the handheld? The trigger and Y adjustment for steering happen here: https://github.com/Valkirie/HandheldCompanion/blob/42691b5f86cd25bdb36262a0531d1f44a0bb6ad3/HandheldCompanion/Managers/MotionManager.cs#L228
2. I always use degrees but also make a habit of noting the unit type in the variable.

I'll try to find the time to write down some more thoughts I have on this subject shortly.

[Yoo-O]

@Yoo-O sounds great! I've been thinking about this on and off myself.

1. If I understand correctly you want to use the position angle change in a relative manner, instead of the absolute position of the handheld? The trigger and Y adjustment for steering happen here: https://github.com/Valkirie/HandheldCompanion/blob/42691b5f86cd25bdb36262a0531d1f44a0bb6ad3/HandheldCompanion/Managers/MotionManager.cs#L228
2. I always use degrees but also make a habit of noting the unit type in the variable.

I'll try to find the time to write down some more thoughts I have on this subject shortly.

@CasperH2O Thanks for your reply. I have tried the following codes: output.Y = InputUtils.Steering(inclination.Angles.X, current.SteeringMaxAngle, current.SteeringPower, current.SteeringDeadzone);

the stick stays in the centre when the device is horizontal which is not user friendly. I want to change the first input parameter "inclination.Angles.X" to "inclination.Angles.X + C", where C is the GRYO angle the time MotionTrigger button is pressed/released, users can define customed centre point in the way, but I don't know how to get the event when MotionTrigger button pressed/released.

[CasperH2O]

Hey @Yoo-O there are a couple of if statements further up in the code that check for "MotionTriggered" like here:

https://github.com/Valkirie/HandheldCompanion/blob/42691b5f86cd25bdb36262a0531d1f44a0bb6ad3/HandheldCompanion/Managers/MotionManager.cs#L184C18-L184C33

and

https://github.com/Valkirie/HandheldCompanion/blob/42691b5f86cd25bdb36262a0531d1f44a0bb6ad3/HandheldCompanion/Managers/MotionManager.cs#L153

Perhaps you can store and or reset a certain value?

I see now also that the sensor fusion is not updated if the button are pressed here:

https://github.com/Valkirie/HandheldCompanion/blob/42691b5f86cd25bdb36262a0531d1f44a0bb6ad3/HandheldCompanion/Managers/MotionManager.cs#L159

Which is the source of the gravity vector angle that we use for this type of steering.

Also, perhaps worse, I don't see it being updated for the steering algorithm. Will investigate, I have some time next Friday.

[Yoo-O]

@CasperH2O Thanks for the advice. But I can't solve the problem of finding the zero point of pitch axis, I decide to give it up. I have tested the KalmanFilter and it can reduce a lot of jitter. Here is the code example may help.

class KalmanFilter2D
    {
        public float[,] Q { get; set; }
        public float[,] R { get; set; }
        public float[,] X { get; private set; }
        public float[,] P { get; private set; }
        public float[,] K { get; private set; }

        public KalmanFilter2D(float[,] q, float[,] r, float[,] initialX, float[,] initialP)
        {
            Q = q;
            R = r;
            X = initialX;
            P = initialP;
            K = new float[2, 2];
        }

        public float[,] Update(float[,] measurement)
        {
            // Prediction
            P = AddMatrices(P, Q);

            // Update
            K = MultiplyMatrices(P, Inverse(AddMatrices(P, R)));
            X = AddMatrices(X, MultiplyMatrices(K, SubtractMatrices(measurement, X)));
            P = MultiplyMatrices(SubtractMatrices(new float[,] { { 1, 0 }, { 0, 1 } }, K), P);

            return X;
        }

        private float[,] AddMatrices(float[,] a, float[,] b)
        {
            int rows = a.GetLength(0);
            int cols = a.GetLength(1);
            float[,] result = new float[rows, cols];

            for (int i = 0; i < rows; i++)
            {
                for (int j = 0; j < cols; j++)
                {
                    result[i, j] = a[i, j] + b[i, j];
                }
            }

            return result;
        }

        private float[,] SubtractMatrices(float[,] a, float[,] b)
        {
            int rows = a.GetLength(0);
            int cols = a.GetLength(1);
            float[,] result = new float[rows, cols];

            for (int i = 0; i < rows; i++)
            {
                for (int j = 0; j < cols; j++)
                {
                    result[i, j] = a[i, j] - b[i, j];
                }
            }

            return result;
        }

        private float[,] MultiplyMatrices(float[,] a, float[,] b)
        {
            int aRows = a.GetLength(0);
            int aCols = a.GetLength(1);
            int bRows = b.GetLength(0);
            int bCols = b.GetLength(1);

            if (aCols != bRows)
            {
                throw new ArgumentException("Matrix dimensions do not match for multiplication.");
            }

            float[,] result = new float[aRows, bCols];

            for (int i = 0; i < aRows; i++)
            {
                for (int j = 0; j < bCols; j++)
                {
                    for (int k = 0; k < aCols; k++)
                    {
                        result[i, j] += a[i, k] * b[k, j];
                    }
                }
            }

            return result;
        }

        private float[,] Inverse(float[,] a)
        {
            if (a.GetLength(0) != 2 || a.GetLength(1) != 2)
            {
                throw new ArgumentException("Matrix must be 2x2 for this inverse method.");
            }

            float det = a[0, 0] * a[1, 1] - a[0, 1] * a[1, 0];

            if (det == 0)
            {
                throw new ArgumentException("Matrix is singular and cannot be inverted.");
            }

            float[,] result = new float[2, 2]
            {
                { a[1, 1] / det, -a[0, 1] / det },
                { -a[1, 0] / det, a[0, 0] / det }
            };

            return result;
        }
    }

Init the filter:

        private static float[,] q = { { 0.1f, 0 }, { 0, 0.1f } };
        private static float[,] r = { { 1, 0 }, { 0, 1 } };
        private static float[,] initialX = { { 0 }, { 0 } };
        private static float[,] initialP = { { 1, 0 }, { 0, 1 } };
        private static KalmanFilter2D kf = new KalmanFilter2D(q, r, initialX, initialP);

Usage:

    case MotionInput.JoystickSteering:
                  {
                      float x = InputUtils.Steering(inclination.Angles.Y, current.SteeringMaxAngle, current.SteeringPower, current.SteeringDeadzone);
                      float y = InputUtils.Steering(inclination.Angles.X - 90.0f, current.SteeringMaxAngle, current.SteeringPower, current.SteeringDeadzone);
                      float[,] measurement = new float[,] { { x }, { y } };
                      float[,] estimate = kf.Update(measurement);
                      output.X = estimate[0, 0];
                      output.Y = estimate[1, 0];
                  }
                  break;

[CasperH2O]

Thank you @Yoo-O , sorry to read you could not get it to work satisfactory.