Test using reducer on _high feed at 1 sec resolution

plane-watch / pw-pipeline

Takes in ADSB information and churns out JSON

https://plane.watch

GNU General Public License v3.0

8 stars 1 forks source link

Test using reducer on _high feed at 1 sec resolution #1

Open tardoe opened 2 years ago

tardoe commented 2 years ago

To "smooth" out the number of plane location-updates on the _high feed, put the _high feed through the reducer mechanism with the ForgetfulSync Map set with a 1 sec timeout.

tardoe commented 2 years ago

I think this is how I'll tackle this:

duplicate the syncSamples DS on pwRouter, create a _low and a _high version with appropriate timeouts.
move the "handle" methods from the worker Into the pwRouter struct to deal with both low and high.

Need more time to think this over.

bluntelk commented 2 years ago

We want to hand out a specific deduped _high feed to pw_ws_broker, and keep sending the full _high feed to clickhouse