search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.91k stars 95 forks source link

Queued Writes #296

Open nickchomey opened 1 year ago

nickchomey commented 1 year ago

rqlite has a great functionality for queueing writes for periodic automatic bulk insertion into the DB. It improves write performance by 15x or more.

Is this something that could be added to LiteFS at some point?

benbjohnson commented 1 year ago

@nickchomey Yes, there are a few places that we could implement batching that would improve throughput. However, the biggest performance issue is going through FUSE. We have plans for implementing LiteFS as a VFS (#165) and that should help performance significantly. The main downside of the VFS is that client applications need to load the extension so it's not quite as easy to use.

nickchomey commented 1 year ago

Oh interesting. I was under the impression that FUSE was chosen because it was more performant.

When you make that switch/implementation that requires loading an extension, would it affect the drop-in SQLite nature of LiteFS (as previously discussed regaring using it with Wordpress)?

benbjohnson commented 1 year ago

I was under the impression that FUSE was chosen because it was more performant.

The main reason for using FUSE was that folks could run on LiteFS without any changes to their application. Writes are fast enough for most use cases (<100 write tx/sec). Reads have the FUSE performance penalty too but only on the first read of a page. After that, the page is fetched from the OS page cache so it's as fast as any other file system.

When you make that switch/implementation that requires loading an extension, would it affect the drop-in SQLite nature of LiteFS?

The VFS extension will be an additional way to use LiteFS rather than a replacement for FUSE. It'll only be needed if you need higher write throughput. The only change you'll need is to be able to run SQLite3::loadExtension() and then you'll need to specify vfs=litefs in the SQLite connection URL.

Being able to inject the extension is definitely the tricky part. That really depends on the application and I'm not sure if Wordpress has a way to run code on the connection before loading the database.

nickchomey commented 1 year ago

Thanks for the details! I'll be happy to help test it all when the time comes