search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

`reindexEncrypted` needs to be crash-resistant #407

Open staltz opened 2 years ago

staltz commented 2 years ago

Context: https://github.com/ssbc/ssb-box2/pull/22

There could be a feature in ssb-db2's reindexEncrypted where it is crash-resistant. Since reindexEncrypted takes no arguments, this is just a matter of storing a boolean in some file (or just the file existence can be the "true" boolean) and then once ssb-db2 starts up it resumes reindexEncrypted if that file exists. Note also that reindexEncrypted does not need to be atomic. Any subset of encrypted records that got decrypted is some partial progress, and then after app restart, the progress can continue.

Also, another thing I notice is that it seems reindexEncrypted is not very efficient, at least not with jitdb. If there are encrypted records at seq 6, 14, 18, 22, and assume max seq is 100 then it'll reindex jitdb from seq 6–100, then reindex 14–100, then 18–100, then 22–100. It could just pick the minimum (6) and do 6–100 once.

arj03 commented 1 year ago

About this:

Also, another thing I notice is that it seems reindexEncrypted is not very efficient, at least not with jitdb. If there are encrypted records at seq 6, 14, 18, 22, and assume max seq is 100 then it'll reindex jitdb from seq 6–100, then reindex 14–100, then 18–100, then 22–100. It could just pick the minimum (6) and do 6–100 once.

This assumes that something comes in and does a query in jitdb between the reindex calls. In jitdb we just reset the indexes so they will be re-created on next query. So if there is nothing concurrent this should only do 1 reindex from 6, right?

staltz commented 1 year ago

Ooooh

staltz commented 1 year ago

Yeah, so let's drop that jitdb optimization idea, you were right.

The "boolean" file in disk to trigger resume of reindexedEncrypted is still going to be relevant to implement.

staltz commented 1 year ago

I got a test written for this, and I got the implementation written correctly, but my implementation isn't passing the tests due to some odd issue with jitdb. Maybe one day we can try having a pair programming debugging.

arj03 commented 1 year ago

Okay, maybe friday afternoon @staltz ? I'm a big hung up right now.

staltz commented 1 year ago

Sounds good. This isn't a blocker right now.