tremor-rs / tremor-runtime

Main Tremor Project Rust Codebase
https://www.tremor.rs
Apache License 2.0
851 stars 126 forks source link

Delay operator #760

Open darach opened 3 years ago

darach commented 3 years ago

Describe the problem you are trying to solve

As a developer/operator of a complex system or service undergoing transformation or migration to a service oriented architecture, a facility to inject delays into streams would be a useful migration/debugging aid for complex system and service migrations.

Describe the solution you'd like

Provide delay operator that can be configured with a configurable default delay in nanoseconds, or a per event delay via metadata ( also in nanoseconds ). Events are delayed either by the default delay, or the per-event delay provided in metadata.

define qos::delay operator mimic_xdc_latency
with
  delay = 30000000 # 30 millis
end;

create operator mimic_xdc_latency;

select event from in into mimic_xdc_latency;
select event from mimic_xdc_latency into out;

Notes

A more advanced implementation would offer variance of the delayed time to a distribution ( eg: pareto ) by some mechanism, but even the basic design highlighted would be super useful for migration/capacity planning and impact analysis for existing systems/services where tremor is in production.

mfelsche commented 3 years ago

Nice idea. And a metadata value that controls the actual delay would also be nice.

darach commented 3 years ago

I clarified the metadata variant in the description - the description wasn't clear enough on per event delay being metadata driven. Thanks @mfelsche !

tobehardest commented 1 year ago

Does this problem still need to be solved? I'd like to try it out if I can.

Licenser commented 1 year ago

Due to the lack of communication on the other issue you asked to be assigned we're holding off on assigning this for now.