val s1 = src.map(f1)
val s2 = src.map(f2)
s1.sumByKey(store1)
.also(s2.sumByKey(store2))
In this case, we will serialize the data from src to a node to run f1 and another to run f2. In fact, storm (and I think heron) support multiple output streams from a node. So we could have a single node run both f1 and f2 and put them on different named outputs. Then the stores subscribe to only one of the two outputs.
This could reduce a storm/heron node in the graph and remove some serializations.
Yes, this will be good to optimize as well. There are a bunch more places where we can eliminate nodes, good to keep track of them via issues. I'll create more.
Consider the case of:
In this case, we will serialize the data from
src
to a node to runf1
and another to runf2
. In fact, storm (and I think heron) support multiple output streams from a node. So we could have a single node run bothf1
andf2
and put them on different named outputs. Then the stores subscribe to only one of the two outputs.This could reduce a storm/heron node in the graph and remove some serializations.
/cc @pankajroark @npraneeth