pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.34k stars 187 forks source link

Fix spilled queries in database #1960

Closed DL6ER closed 1 month ago

DL6ER commented 1 month ago

What does this implement/fix?

Improve query storing algorithm to better cope with bursts of queries of arbitrary size and frequency. See related Discourse bug chasing for further details.

This is overall an improvement as the new strategy guarantees that no queries can be lost on one hand and drastically reduces the workload on the other hand by using persistent prepared statements instead of compiling them once per second.


Related issue or feature (if applicable): N/A

Pull request in docs with documentation (if applicable): N/A


By submitting this pull request, I confirm the following:

  1. I have read and understood the contributors guide, as well as this entire template. I understand which branch to base my commits and Pull Requests against.
  2. I have commented my proposed changes within the code.
  3. I am willing to help maintain this change if there are issues with it later.
  4. It is compatible with the EUPL 1.2 license
  5. I have squashed any insignificant commits. (git rebase)

Checklist:

pralor-bot commented 1 month ago

This pull request has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/discrepancy-between-pi-hole-api-client-query-count-and-logs/70065/17

DL6ER commented 1 month ago

Confirmed working by letting this run locally for 2 days by now and independent confirmation by OP on Discourse:

https://discourse.pi-hole.net/t/discrepancy-between-pi-hole-api-client-query-count-and-logs/70065/19