Follow last valid result, multiple camera, kalman filter

[bnogalm]

I was implementing something similar some years ago, by using dlib

I lost the code, so i tried this one that seems to work. Good work!

I miss some stuff i implemented that improved the performance and helps when someone cross the area looking what you are doing.

For the face area recognition, lookup starting by last valid result, and move around. To look around I was starting by using last displacement orientation vector, as a estimation.

Another is to use a kalman filter to smooth the result signal.

On the other hand I give up developing it because i was unable to run multiple ps3 eye cameras, if it would be possible now with this driver you use, same code could run to use an array of cameras and track face by using closer ones and estimate error, knowing the distance between cameras.

If i find time i could implement it. But i have lot of delayed work in other repos... so i write some ideas.

[AIRLegend]

Hey, thanks for the feedback!

The "last face position" idea could be a nice addition. Certainly, it could be useful for solving the "two face on camera" problem. Feel free to add it if you want. It shouldn't be too difficult :)

Regarding the Kalman filter, I explored the idea quite long ago, but it was not easy because of the prior knowledge about the noise distributions (would vary between cameras/setups). Instead, I added a moving average on the facial keypoints and leveraged the already available signal filters on Opentrack. For me, at least, Kalman filter has been the worst of them :/

[searching46dof]

Actually, using multiple cameras for 360deg coverage wouldn't be difficult if implemented as a separate server processing multiple udp input streams from multiple instances of AItrack.

• The coordinates and orientation of each camera would need to be configured relative to a designated primary camera.
• Upon clicking start, each camera would need to identify multiple faces in the camera's view and associate or add them to a list of logical faces with references (in case tracking is lost). An association with an logical face from a different camera is based on matching translated x,y,z coordinates (base on configuration relative to primary camera)
• Upon losing tracking of a face, the reference count of the corresponding logical face is decremented. When the reference count reaches zero, the logical face is deleted from the list.
• For each logical face, an average of the translated coordinates from each camera is performed.
• If a single output stream is provided, the simplest solution is the center weighted face of the primary camera.
• If multiple output streams are provided, then multiple faces in the primary camera need to be associated with logical faces based on proximity to multiple points in the preview window.
• It would also require disabling the Z axis scaling in AItrack's PositionSolver::solve_rotation(...) for the camera's coordinate translation and re-applied after the logical face coordinate averaging: // We dont want the Z axis oversaturated since opentrack has +/-600 centimeter range face_data->translation[2] /= 100;

Regarding the Kalman filter, smoother results with better filter performance may be obtained by using an Unscented Kalman Filter. Prior knowledge regarding the noise from each camera may not be necessary. The noise would depend on the resolution of each camera, distance to the face, ambient lighting and the jitter in the measurements. Perhaps a logarithmic slider bar for each kalman filter instance per coordinate (x,y,z,yaw,pitch,roll) per face with a maximum face limit. Maybe three (left,center,right) in the primary preview window.