d2iq-archive / serenity

Intel:Mesosphere oversubscription technologies for Apache Mesos
Other
69 stars 21 forks source link

Refactor data transformation between components in pipelines. #86

Open bwplotka opened 9 years ago

bwplotka commented 9 years ago

Currenlty we have not flexible solution. We are reusing some double fields in ResourceUsage (e.g net statistics) for EMA puposes. For better code readability we should reuse the same fields and add some Tags in which we will write what filtering were done on given data. e.g: Source -> RU with perf instructions,tag<> -> EMA -> RU with EMA instructions, tag<ema> Using aforementioned mitigation enables for us the possibilty to configure pipeline even without recompling.

To elaborate about tagging, here's use case: "As a serenity module developer I would like to register for ResourceUsage information from two different sources and I would like to know which message is from which filter (eg, raw, ema)". It's about adding distinguishable labels to non-distinguishable messages. I'm reluctant to it, because it leads to hardcoded pipelines - but it's a idea that comes from time to time (but I haven't seen any real "i need this" yet).