search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.82k stars 93 forks source link

Litefs secondary replication position #301

Closed tjheeta closed 1 year ago

tjheeta commented 1 year ago

I was testing out litefs on my application. There is a workflow where a new page is created and the user is re-directed to that page. I found that occasionally after a write that the secondary would not have the data causing a 404 on the redirect.

I was about to implement a wait state like fly_rpc_postgres, however, the "pos" file doesn't appear to have information that can be used for this purpose. I was looking for a monotonically increasing transaction id that can be compared on the replicas to the master.

Is there anything that can be used for this purpose?

tjheeta commented 1 year ago

Looking a bit closer at the "pos" file, it looks like the data is there.

${transaction_number}/${db_hash}
0000000000000060/d75c6c2c863d0683

Is the first number the transaction id?

tjheeta commented 1 year ago

Well, if it isn't the transaction id, I'm not sure how I got the rpc wait functionality running. So closing.

benbjohnson commented 1 year ago

Yes, the first part is the TXID and the second part is the database hash. They are both formatted as hex-encoded uint64 values, separated by a slash, and ending with a newline.

There's a proxy server (https://github.com/superfly/litefs/pull/271) that's coming in the next release (v0.4.0) that will automatically redirect writes to the primary and track the last written TXID so you can read-your-writes from replicas.