cda-group / arcon

State-first Streaming Applications in Rust
https://cda-group.github.io/arcon/
Apache License 2.0
175 stars 17 forks source link

Async Sources #281

Open Max-Meldrum opened 2 years ago

Max-Meldrum commented 2 years ago

As of now, source polling is not async.

pub trait Source: Send + 'static {
    type Item: ArconType;
    /// Poll Source for an Item
    fn poll_next(&mut self) -> SourceResult<Poll<Self::Item>>;
    /// Set offset for the source
    ///
    /// May be used by replayable sources to set a certain offset..
    fn set_offset(&mut self, offset: usize);
}

My guess is that most sources that we and others implement will be async/await in Rust.

By having it async and using cooperative scheduling on a single core between a Source and operator(s), we will utilise the CPU much better compared to a blocking approach.

Related to #277