search

poundifdef / SmoothMQ

A drop-in replacement for SQS designed for great developer experience and efficiency.
GNU Affero General Public License v3.0
1.7k stars 30 forks source link

Improve speed #6

Open nicois opened 5 days ago

nicois commented 5 days ago

Nice work. Coming here via HN.

There have been a few examples on HN on how to dramatically speed up sqlite access with a few configuration tweaks. I've encapsulated these in https://pkg.go.dev/github.com/nicois/fastdb . Looking at your code, I think you would support significantly greater throughput under heavy load when using this approach, splitting our your SELECT calls from mutating database calls.

I can make a PR if you like, using the above module, or you might choose to implement these few settings yourself directly. The key settings can be seen here: https://github.com/nicois/fastdb/blob/main/main.go#L97

poundifdef commented 1 day ago

If you can share some before/after benchmarks on this repository then I'd be happy to take a closer look. I'm not sure that all of these settings make sense. Nearly every operation is a write (enqueue/dequeue). I'm not familiar enough with the durability of different synchronization pragmas, so I'd be interested to think through what is "safe enough" in practice.