Off-center camera frame of reference causes large axis bleed-through

[IslandJohn]

Describe the bug

I'm trying to use AITrack/Opentrack with a Logitech Brio (FOV 90) on a large display (55") at a viewing distance of about 30-32" (for context, this is a flight simulator set up that gives a 1:1 FOV for real life and sim). With this setup, the camera is pointing down by more than 20 degrees in order to center my face at the origin.

This results in incorrect tracking because of the initial frame of reference; for example:

• Full head yaw left right (about 30 degrees), shows a pitch and roll change of about 15 degrees.
• Z translation causes both pitch and Y changes
• X translation causes yaw changes

Is it possible to compensate for this? Given the FOV (?), distance, and initial rotation, can these be canceled out?

It seems like the model assumes a perfectly centered camera, so maybe it's possible to apply these corrections.

To Reproduce

Place camera at a high enough position that there's substantial rotation (say 15 degrees) to center view.

Offset camera left right to make the issue even more pronounced.

Expected behavior

The expectation is that orientation of the camera could be configured/tweaked (beyond just FOV and distance).

Could there be some (maybe advanced) configuration added to describe the initial setup of the camera?

Environment (please complete the following information):

• Operating system version 19044.2364
• Webcam model: Logitech Brio
• AITrack version: 0.7.1

[IslandJohn]

I've thought about this a bit in the ideal case (i.e. camera directly in front of you). Even in such a scenario, a translation along X (with head pose maintained forward), should create artificial yaw (since the model would interpret seeing the side of the face as yaw). Or, similarly Y/pitch. Since you know the FOV this can be corrected (e.g. add/subtract atan X/dist - or Z+dist).

I'd be curious if this correction would work as I suspect, so I may poke at it, though I may be out of my depth (if you have any pointers on where to hack this in, it would be helpful).

Rotation of the camera frame seems much more complicated and I haven't thought it about it yet.

What are your thoughts about trying to correct for all of these type of cases?

[AIRLegend]

Hi!

Yeah, you're right. Currently "roll" is not compensated in such cases. However, yaw with lateral offset and pitch with vertical offset actually are. (https://github.com/AIRLegend/aitrack/blob/master/AITracker/src/PositionSolver.cpp#L323)

Marking the issue as an idea for improvement. In the meantime I'd suggest tweaking the curves in Opentrack adding a deadzone so the effect is mitigated.

Thanks :)

[IslandJohn]

Wow, nice, so the ideal case if already working.

I'll think through correcting the translation based on my camera pitch angle, and see if I can hack something in to try it out.

Any ideas why the head yaw is also causing roll in my case? Looking at the preview, this doesn't seem normal.

[searching46dof]

What you are describing is parallax from the camera's perspective. It would require multiple trackers to accurately compensate for these symptoms. For a single camera, it would need to be placed in the center of the tv but that would obstruct the view of the tv. Otherwise it would require an accurate initial position/orientation of the camera and person each time starting up aitrack/opentrack.

You can mitigate the symptoms by:

• center camera (lef/right) position relative to the tv.
• orient the camera so that you are in the center of the preview window.
• enable opentrack option "center at startup" and look at the center of the tv when starting aitrack and opentrack
• use a 1:1 scaling factor for x, y, z and roll mapping and limit them since you are simulating sitting in a plane cockpit (or racecar). These symptoms minimally affect the 1st person perspective of a flight or driving simulator where you will be going back to a zero position. Or you can completely disable x,y,z and roll entirely which is less realistic but may provide a better gaming experience.

This is more of a major problem for VR tracking when physically moving around a play area.

[IslandJohn]

Thanks, I see the parallax for the for X/Y translation and pitch/yaw, but do you have any thoughts on what is inducing the roll? Translation to, say, bottom right shouldn't cause any roll to be detected.

I'm hoping to replace TrackIR (which also suffers from this, but since the reflectors are on top of a hat it's less noticeable - however TrackIR has other issues so I would prefer to switch to this solution).

Maybe I can calculate the corrected translation by simply taking the original translation and rotating it based on the camera setup. Then I have to correct for the for the induced pitch, and this seems to be an offset based on the angle from the camera to the original position.

Before I go down this hunt, I'm trying to understand the roll issue. I do use all 6 DOF while flying (e.g. looking over the cowling, looking at switches, looking around a window pillar in a turn etc., which use translation and roll).

[searching46dof]

Roll is tilting the head to either side. In an ideal situation, the camera would be placed at the same height as the face. Since it is placed above or below the monitor any head yaw would also show a head tilting which would be interpreted as roll.

For smaller monitors, there is less of a perceived roll from the camera's perspective. For larger tv's there is a larger perceived roll unless you sit farther away.

Note that there is currently an issue with detection farther than 2 meters (https://github.com/AIRLegend/aitrack/issues/173) which seems to be related to optimizing the size of the bounding box for face detection.

This image from https://user-images.githubusercontent.com/107732149/210458789-8bca6b16-b1eb-4b99-be48-996718f8fd5c.png should make the roll issue more visible. In the picture for roll, tilting the head to right is positive roll while tilting the head is negtive roll. But from an elevated camera's perspective, positive yaw to right will be perceived as negative roll and negative yaw to left will be perceived as positive roll.