Remove Outdated DP from Queue if not already processing

JulianKropp / stream_pipeline

Modular pipeline for processing data streams and measuring each step

MIT License

0 stars 0 forks source link

Remove Outdated DP from Queue if not already processing #26

Closed JulianKropp closed 2 months ago

JulianKropp commented 2 months ago

3ff4fb49c53457ebec26b3bf21a725c9e7edf493

JulianKropp commented 2 months ago

It doesnt make sense to implement a OUTDATED_PROCESSED. Becuase dp in a queue can not race anouther dp which is also in the queue. Maybe in the future. But not now.

Idea:

A nice way would be to stop the processing if a newer dp end the controller. But a nice way to implement this would be a pubsub system inside the pipeline so the modules can communicate and also the controller can send a stop to the current module