search

funkey / recept

Pen event interception for line smoothing on the reMarkable 2 tablet
MIT License
163 stars 13 forks source link

[Improvement] Implement Kalman Filtering #7

Open aussetg opened 3 years ago

aussetg commented 3 years ago

The task of estimating the correct point at time T given past times is called Filtering in statistics. This is exactly what's happening here so it's the perfect fit :) Should eliminate the input lag (provided computations are not too slow).

https://en.wikipedia.org/wiki/Kalman_filter

Just putting it here as reminder, I may do it myself if I manage to find in which part of my brain I lost all my C++ knowledge.

roborros commented 3 years ago

aussetg, actually, I don't fully agree. The Kalman filter could definitely be used here but it has its ups and downs - mainly because the KF's core idea is built around predicting the next step which is hard in this case.

1) KF can completely eliminate the lag only in the special case when you can measure the input to the system and have a good enough model of the system. In this case, the input is your mind and the system your hand - meaning no prediction possible.

2) You could build the KF around a model of a spline or a similar shape and try to predict where will the future points land. That would effectively reduce the lag compared to the convolution filter used by funkey's solution but at the cost of other problems in the cases when your model fails to model the reality (e.g. sharp direction changes and line ends). In practice that means lower lag but slightly lower precision.

In my opinion, the best way to go is to use some post-processing to smooth the points after they are drawn - that is what OneNote and Whiteboard do and I think the results are great.

funkey commented 3 years ago

Fully agree on the trade offs of using a Kalman filter. By the way, you might want to check out the kalman branch of this repository where I tried exactly that.

In summary: For straight lines (where the Kalman filter conditions are fulfilled) you will indeed observe less latency. In this situation, you could even beat the latency of a vanilla ReMarkable. However, this comes at the expense of prediction errors if you change direction rapidly (like at the top of a lower-case "l"). Here, the Kalman filter will overshoot. You can tweak around with the model parameters to find a balance between latency and accuracy, but eventually I figured the solution is no better than the simple mean filter in the main branch.

Smoothing points after the fact is definitely the way to go. Unfortunately, we'd need xochitl's source code for that...