Database performance drops significantly over time

shenef commented 3 years ago

I've been using InputScope on my work pc to track my activity across the day but eventually had to stop using it cause i couldn't access the data anymore. Full process:

my mouse dataset has 14 million entries
trying to access it the program seems to try to load the whole database into the RAM while using 25% of the CPU (utilizing only a single core?)
it manages to allocate about 2.5GB of RAM and then the page stops loading with the following error:
A server error occurred. Please contact the administrator.

Before that it already slowed down, taking up to 2 minutes to load the page. Current Database size is 916MB. A part of the problem can be attributed to the machine since it uses an older 4 core cpu and a HDD but i'm fairly certain that this still happens on current hardware.

suurjaak commented 3 years ago

Indeed, the statistics pages of the current version do not scale for long-term use.

The next release will handle large datasets better.

suurjaak commented 3 years ago

Added event limits to statistics pages, to ensure page load does not hang or run out of memory.

Can you confirm it works better now, with the new v1.3 release?

shenef commented 3 years ago

It works fine now, page loads are much faster and don't crash anymore. Even re-using the old 920+MB database works fine. Using the repack feature also reduced that db to around 850 MB iirc.

suurjaak commented 3 years ago

Excellent, closing the issue.

suurjaak / InputScope

Database performance drops significantly over time #5