search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.82k stars 93 forks source link

Improve deletion via LTX zero commit #382

Closed benbjohnson closed 11 months ago

benbjohnson commented 11 months ago

Currently, LiteFS communicates database deletion to replicas by sending a "dropdb" frame in the stream. However, this doesn't work well for LiteFS Cloud as it acts as the data authority and doesn't receive a stream. LFSC also needs to retain old copies of the database in case of recovery in the future.

This issue is to track the integration of "zero commit" LTX files (https://github.com/superfly/ltx/pull/46) into LiteFS & LFSC.

gc-victor commented 11 months ago

also needs to retain old copies of the database in case of recovery in the future

Sorry if this is too obvious, but I want to highlight this case.

In my use case, a new database can be recreated with the same name as a deleted database, but it has no relation to the old one except for using the same name.

Case: DB X is created DB X is deleted -> it is stored as a deleted DB DB X is created (without any relation with the old DB other than the name) DB X is deleted -> it is stored as a deleted DB

benbjohnson commented 11 months ago

The filename is the only identifier for the database in LiteFS so the two databases are related in that sense. On the LFSC side, you'd still see an entry for the database after deletion but it wouldn't show up on the LiteFS side if the database size was set to 0 in the most recent transaction.

gc-victor commented 11 months ago

If I delete and recreate a database with the same name, will it cause any problems? And if I delete it again, will it affect the previous deletion?

benbjohnson commented 11 months ago

No, it shouldn't cause any problems. With LiteFS, a transaction represents a specific set of bytes that make up the database at a given point in time. By making a zero-length database represent a deletion, it's just another transaction from LiteFS' perspective.

So internally, the TXID will keep incrementing even though you've created/deleted multiple times. e.g.

benbjohnson commented 11 months ago

@gc-victor I implemented soft delete so you can actually delete databases while connected to LiteFS Cloud (https://github.com/superfly/litefs/pull/395). You can test it using the Docker image for the PR: 

COPY --from=flyio/litefs:pr-395 /usr/local/bin/litefs /usr/local/bin/litefs
gc-victor commented 11 months ago

@gc-victor I implemented soft delete so you can actually delete databases while connected to LiteFS Cloud (#395). You can test it using the Docker image for the PR:

COPY --from=flyio/litefs:pr-395 /usr/local/bin/litefs /usr/local/bin/litefs

Great! I will test it next week.