search

jovandeginste / workout-tracker

A workout tracking web application for personal use (or family, friends), geared towards running and other GPX-based activities
Other
945 stars 31 forks source link

Feature/batch matching #262

Closed pixelspark closed 3 weeks ago

pixelspark commented 4 weeks ago

This might help alleviate #261. This changes the matching in the background loop to fetch segments and workouts in batches. This is a bit easier on the database (it uses smaller queries) and should scale a bit better.

jovandeginste commented 3 weeks ago

Did you test this with your database?

jovandeginste commented 3 weeks ago

However, in this case, I'm wondering if just reducing the Limit(1000) in L184 would be enough to get rid of the outer loop of your PR? The point is that the background loop will come back every minute and process the next X segments.

So basically, I mean a hybrid of the original code with your fetch in batches of the workouts, and reduce the Limit from 1000 to 10 or so.

pixelspark commented 3 weeks ago

Did you test this with your database?

So I run my 'production' instance on a NAS. This has about 600+ workouts. I tested on my laptop (M1 Max) using a copy of the database. My laptop however is way faster than the NAS, so I am not entirely sure the issue is fixed (especially the 'cannot start transaction inside transaction' thing seems like a deeper error to me). However the batching does seem to lead to less 'expensive' queries from being generated (and should at least reduce peak memory usage as not all workouts need to be loaded at a time for matching).

However, in this case, I'm wondering if just reducing the Limit(1000) in L184 would be enough to get rid of the outer loop of your PR? The point is that the background loop will come back every minute and process the next X segments.

So basically, I mean a hybrid of the original code with your fetch in batches of the workouts, and reduce the Limit from 1000 to 10 or so.

Yes, that would work, will change it.

As for the other comments, I will reply to those individually.

jovandeginste commented 3 weeks ago

Curious, I can't seem to run the linter here; can you run it locally?

pixelspark commented 3 weeks ago

Curious, I can't seem to run the linter here; can you run it locally?

After the last commit:

image

jovandeginste commented 3 weeks ago

image

jovandeginste commented 3 weeks ago

Apparently, I need to add a .golangci.yml to the project; I will add this in a later PR (and fix any issues that come from it).