cda-group / arcon

State-first Streaming Applications in Rust
https://cda-group.github.io/arcon/
Apache License 2.0
175 stars 17 forks source link

ArconState/Index/Backend Rework #303

Open Max-Meldrum opened 2 years ago

Max-Meldrum commented 2 years ago

The existing design is not the best as it currently exposes functions that the user should not have access to.

ArconState

set_key / set_timestamp are exposed to users. This is something that only the internal runtime should be able to modify/call.

Backend

For the state constructor, users should not have access to Backend itself, but some form of wrapper around it with limited exposed functions.

struct WriteContext {
   key: u64,
   timestamp: u64
}
pub struct BackendContainer<B: Backend> {
   crate context:  WriteContext,
   crate inner: Arc<B>
}

OperatorBuilder {
  operator: Arc::new(|| ...),
  state: Arc::new(|c: BackendContainer| ...), // Here instead of Backend
  conf: Default::default(),
}

Index

Right now indexes do not have an explicit constructor function and all of them expect Arc<Backend> input. The following might be a good option instead.

pub trait IndexOps {
    fn create(id: &str, container: BackendContainer) -> Self;
    ....
}