Add persistence

[ryardley]

Actix actually provides a great way to manage dependency injection within the actor model which prevents us from having to utilize impl Traits and dependency injection. Instead we can use the Recipient<M> type to act as an injection point.

Provide the data actor as a writer and a reader.

• [ ] The write should by typed Receiver<Insert>
• [ ] The reader should be typed Receiver<Get>

This will enable us to easily continue to mock a memory only data store in tests.

One idea might be to pass the actor around as a tuple struct. The benefit here is we can now use this to test with a memory only actor in the background.

#[derive(Clone)]
struct Database(Recipient<Get>, Recipient<Insert>);
impl Database {
    pub async fn read(&self, msg:Get) -> Result<Option<Vec<u8>>>{
        Ok(self.0.send(msg).await?)
    }

    pub fn write(&self, msg:Insert) {
        self.1.do_send(msg)
    }

    // use this for testing
    pub fn from_in_mem(&data_addr: Addr<InMemDb>) -> Self {
      let d = data_addr.clone();
      Self(d.recipient(),d.recipient())
    }

    // use this for production 
    pub fn from_sled(&data_addr: Addr<SledDb>) -> Self {
      let d = data_addr.clone();
      Self(d.recipient(),d.recipient())
    }
}

both of these can be injected into the Keystore actor if it takes Database:

pub struct Keyshare {
    fhe: Arc<Fhe>,
    data: Database,
    bus: Addr<EventBus>,
    address: String,
}

The benefits of this pattern:

1. Streaming writes which wont block.
2. Testable mock-able interface

[ryardley]

There is a broader question here about in memory-db-lifecycle here. We need to be able to resuscitate every actor's state from the db in the event of a failure - every actor state change should be lazily saved as the last thing it does after processing a message. (ie send it's data change to the Database) - it might be worth saving the last message that was processed too to be able to process a buffer of messages but that would be more a hardening exercise.

Actor's would need to be IDs aware (eg "{e3_id}/keyshare/pubkeyshare" etc.) - i think they already are?. The router would store the keys of the current e3_id context hashmap of e3_ids from which it is possible to attempt load the child actors after a restart.

I kind of think that we can start with setting up a test for a clean restart between operations - it seems like it would be advantageous for this to have some kind of process manager like https://github.com/gnosisguild/enclave/issues/64 for testing - otherwise we have to run pkill or something which is possible but messy.

[ryardley]

Was thinking we could further break this task down:

• [ ] Create the above DataStore tuple struct for injection
• [ ] Hydration unit tests for the following using the in-memory struct:
  • [ ] Keyshare stores secret keyshare and publickeyshare
  • [ ] Router stores a list of e3_ids
  • [ ] PublickeyAggregator stores it's state after every message
  • [ ] PlaintextAggregator stores it's state after every message
  • [ ] Sortition stores it's state every add / remove
• [ ] New restart integration test for nodes to ensure they are persisting their state on restart.

By hydration I mean implement a fn started() on the actor with a ctx.wait that loads it's state from the database if it exists. This might be a term from web development when data is hydrated to the client upon which it is made 'alive' into components.

@nginnever thoughts?

[ryardley]

Key structure?

• Keyshare - v1/{e3_id}/keyshare
• CommitteeMeta - v1/{e3_id}/meta
• FheParams - v1/{e3_id}/fhe
• Router - v1/router
• PublickeyAggregator - v1/{e3_id}/publickey_aggregator
• PlaintextAggregator - v1/{e3_id}/plaintext_aggregator
• Sortition - v1/sortition/ (will need nodes and possibly committee cache)