[Bug] sqlite WAL never checkpointed, leading to constantly-increasing disk usage

EtaoinWu commented 3 weeks ago

Is this a support request?

[X] This is not a support request

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

I have a ~20 node tailnet running and my sqlite database is around 500KB. After running it for several months, I realized that it's almost occupying 1GB of disk space now. Upon inspection, I found that my db.sqlite-wal is growing at a steady 500KB/hour, which equals >4GB/year. Headscale never performs any checkpointing operation, and its wal_autocheckpoint was set to zero here. This lead to the ever-growing WAL file.

Expected Behavior

There should be some periodic checkpointing of the sqlite database, either by time or by number of transactions.

Steps To Reproduce

Set up a tailnet with sqlite
Add a bunch of nodes, keep them connected and let them ping each other periodically
Wait for months
See that db.sqlite-wal is huge

Environment

- OS: `Ubuntu 20.04.6 LTS`
- Headscale version: `v0.23.0`
- Tailscale version: `1.72.0`

Runtime environment

[X] Headscale is behind a (reverse) proxy
[X] Headscale runs in a container

Anything else?

No response

aalmenar commented 3 weeks ago

In just a matter of two weeks, with personal usage, wal file is now 63MB from a fresh install.

nblock commented 3 weeks ago

A workaround to recover diskspace and write the content of the WAL back to the main database:


$ sudo systemctl stop headscale
$ sudo -u headscale -i
$ sqlite3 /path/to/headscale/db.sqlite VACUUM
$ sudo systemctl start headscale

juanfont / headscale