search

zhaofengli / attic

Multi-tenant Nix Binary Cache
https://docs.attic.rs
Other
965 stars 73 forks source link

sqlite backend extremely slow on zfs #113

Open yorickvP opened 7 months ago

yorickvP commented 7 months ago

I was getting extremely slow speeds with the default sqlite backend on zfs, I switched to postgres and got a 24x upload rate speedup. I was also hitting https://github.com/zhaofengli/attic/issues/24 occationally.

Could there be an option to turn off fsync on the sqlite backend? That should speed it up massively.

nazarewk commented 6 months ago

this is the same as:

Mic92 commented 5 months ago

This is how I work around in atuin: https://github.com/atuinsh/atuin/issues/952#issuecomment-1801308296

srd424 commented 5 months ago

I'm seeing pathological slow speeds and timeouts even on ext4 - my underlying storage stack is a bit of a muddle, but shouldn't be anywhere as bad as I'm seeing. I switched to hosting the database on a small ext2 LV and things are a bit better, though not a lot.

I'm beginning to suspect that there's something about attic's access patterns that just interacts really badly with sqlite. I tried switching the database to WAL journalling but that produced a negligible improvement if any. Using eatmydata to neuter fsyncs is the only thing that really helps - that or sticking the database on a tmpfs :)