infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
https://www.fluvio.io/
Apache License 2.0
3.79k stars 500 forks source link

Improve the load of store specs on local clusters #4177

Open fraidev opened 2 weeks ago

fraidev commented 2 weeks ago

Based on https://github.com/infinyon/fluvio/pull/4175#discussion_r1755921039 we should improve how we are loading store specs.

For now, we're creating or loading stores on the fly, like a cache:

fn get_store<S: Spec + DeserializeOwned>(&self) -> Result<Arc<SpecStore>> {
    let key = S::LABEL;
    let read = self.stores.read();
    Ok(match read.get(key) {
        Some(store) => store.clone(),
        None => {
            drop(read);
            let mut write = self.stores.write();
            let store = Arc::new(SpecStore::load::<S, _>(self.path.join(key))?);
            write.insert(key, store.clone());
            drop(write);
            store
        }
    })
}

@sehz said that a better approach would be to load all the specs first and remove this write lock from the get_store.