search

cjekel / piecewise_linear_fit_py

fit piecewise linear data for a specified number of line segments
MIT License
289 stars 59 forks source link

How to prevent poor fitting #111

Open dsandber opened 9 months ago

dsandber commented 9 months ago

The image below shows a fit I got using pwlf. Adjacent linear segments are shown in alternating colors for easy visualization. The blue dots are the original data points.

Is this issue not enough points in that region? Any suggestions on how to improve this?

image

Thanks!

cjekel commented 9 months ago

This looks like a failure in the optimization. Can you try fit fast with a large population. Maybe pop=50. The runtime might be quite a bit longer.

How many line segments do you have? It's possible you have too many to be a well posed problem.

On Mon, Sep 11, 2023, 12:33 dsandber @.***> wrote:

The image below shows a fit I got using pwlf. Adjacent linear segments are shown in alternating colors for easy visualization.

Is this issue not enough points in that region? Any suggestions on how to improve this? [image: image] https://user-images.githubusercontent.com/5387753/266991252-c130e5e5-91da-46e8-a396-198453c337d5.png

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/cjekel/piecewise_linear_fit_py/issues/111, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJ5Z4NHHGRLSBO2YA5VBLLXZ3SJLANCNFSM6AAAAAA4TDIV7Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

dsandber commented 9 months ago

The knot-points are fixed, so there's no optimisation if I'm understanding you right. The knot points have to be fixed because I'm optimising for an absurdly CPU constrained microcontroller. There's 128 segments. Seems like I should first fit a curve somehow, and then use pwlf after that once the data is continuous. Any suggestions for what to use to fit the curve? The underlying curve is very smooth with only slow changes in slope.

Update: Seems I want a smoothing spline. And then I'll feed that to PWLF.

cjekel commented 9 months ago

Are you running into something like this? https://jekel.me/piecewise_linear_fit_py/examples.html#bad-fits-when-you-have-more-unknowns-than-data

If you don't need 128 segments, i would reduce the number of segments down to a dozen or so fixed break points. As you said, it's not that non linear of a relationship.

If you need to get artificially more data, fit a linear interpolation model to the data. Then sample from that model. Scipy and numpy have these linear interpolation schemes. There is also linear kernel rbf.

On Tue, Sep 12, 2023, 11:37 dsandber @.***> wrote:

The knot-points are fixed, so there's no optimisation if I'm understanding you right. The knot points have to be fixed cause I'm optimising for a very CPU constrained microcontroller. There's 128 segments. Seems like I should first fit a curve somehow, and then use pwlf after that once the data is continuous. Any suggestions for what to use to fit the curve? It's a very smooth with only slow changes in slope.

— Reply to this email directly, view it on GitHub https://github.com/cjekel/piecewise_linear_fit_py/issues/111#issuecomment-1715365686, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJ5Z4PH3T3VCGLSQXDWTRLX2AUNBANCNFSM6AAAAAA4TDIV7Y . You are receiving this because you commented.Message ID: @.***>

dsandber commented 9 months ago

In the process of making an example for you I realised I had a bug that was causing the issue, however after fixing it there's still a problem.

I have 256 data points and 129 knots. I've attached an example with the data as a json. Running it should open the graphs in a browser. Seems like the gaps are causing an issue even though there's more data points than segments.

The graphs look like:

image

data.txt example.txt