Facemesh is run asynchronously in a Web Worker. It's sent camera frames, does its processing, then returns the face location predictions, but by the time it's processed it, the world has moved on: there have been multiple camera frames, and points tracked on the image have moved, moving the pointer (it doesn't wait for facemesh; it uses point tracking which is faster).
The goal is to place (and remove) tracking points on the image based on the face location, but the face location information we get is for an outdated camera frame.
Now, this generally isn't that much of a problem. It causes some jitter due to points being added and removed stupidly, but the software still generally tracks your head movements.
It can be a huge issue though, if hardware acceleration is disabled, if I recall correctly?
Ideal Solution
Make facemesh fast enough to run it synchronously.
Invalid Solution
First I tried moving the facemesh result, translating it by the averaged movement of the tracking points during the time it took facemesh to return results. That kind of.. almost works? But it's sort of backwards....
This doesn't account for face rotation, or visual size change (movement towards/away from the camera)
Part of the point is to update which points on the image should be tracked. If these are bad, or don't exist, it doesn't work!
Complex Solution
In order to compensate for the latency of results from the Worker, I made a "time travel" system:
Recording camera frames since the frame sent to the worker for processing
Picking points to add, based on the outdated frame
Playing back the recorded frames, tracking movement for the new points, to bring them up to date with the current frame
...but it was too slow to actually do the replaying. Tracking the points is fast enough to do in realtime, but when time is bunched up like this, it causes a lot of lag, and makes it unusable.
To make this work I would need to optimize it.
Dedupe grayscale() computation...
WebAssembly for tracking points?
And I would need to support time travel for adding AND removing points
Simple Partial Solution?
To be honest I don't remember if I implemented this in order to address this or not, but I feel like I made the adding and removing of points more strict, more constrained to the facemesh result, and even though the result is outdated, this actually helped a lot?
This isn't a huge issue at this point, since:
The points tracked on the image are what I actually use to move the pointer, and
Once your head comes to rest, it quickly corrects the tracking points.
And the faster your head is moving, the harder it is to notice inaccuracy in tracking.
As long as the hardware is good enough?
Look I worked on this a long time ago, I don't remember the situation.
Facemesh is run asynchronously in a Web Worker. It's sent camera frames, does its processing, then returns the face location predictions, but by the time it's processed it, the world has moved on: there have been multiple camera frames, and points tracked on the image have moved, moving the pointer (it doesn't wait for facemesh; it uses point tracking which is faster). The goal is to place (and remove) tracking points on the image based on the face location, but the face location information we get is for an outdated camera frame.
Now, this generally isn't that much of a problem. It causes some jitter due to points being added and removed stupidly, but the software still generally tracks your head movements. It can be a huge issue though, if hardware acceleration is disabled, if I recall correctly?
Ideal Solution
Make facemesh fast enough to run it synchronously.
Invalid Solution
First I tried moving the facemesh result, translating it by the averaged movement of the tracking points during the time it took facemesh to return results. That kind of.. almost works? But it's sort of backwards....
Complex Solution
In order to compensate for the latency of results from the Worker, I made a "time travel" system:
...but it was too slow to actually do the replaying. Tracking the points is fast enough to do in realtime, but when time is bunched up like this, it causes a lot of lag, and makes it unusable.
To make this work I would need to optimize it.
Simple Partial Solution?
To be honest I don't remember if I implemented this in order to address this or not, but I feel like I made the adding and removing of points more strict, more constrained to the facemesh result, and even though the result is outdated, this actually helped a lot?
This isn't a huge issue at this point, since:
As long as the hardware is good enough? Look I worked on this a long time ago, I don't remember the situation.