First of all, this is actually quite a simple piece. Especially if you first read the diff that is not automatically expanded by GitHub (i.e. everything but dsl.h and test.cc), then test.cc, and only then get to the very dsl.h.
The main purpose of this code is to get rid of the error-prone manual specifications of InputOf<>/OutputOf<>/SinkOf<>, which I used a while back to just get the test running. The secondary, yet important, purpose of this code is to support parallel execution (which is as simple as managing the "input available until / input processed until" trackers, as I wrote about here).
The other, less important goals, are:
Readability of the pipeline-defining code. You'll see it right away.
Extensibility with respect to adding monitoring / dashboard / controls.
Ultimately, the within-machine node running its own pipeline would need a Karl-like status page. With a) tracking queue size (and what is stuck), b) tracking performance of each individual worker, and c) controls to change parameters on the fly, most notably, the processing block sizes per module (for the throughput / latency tradeoff).
These are all in some distant-ish future. For now, it's just the by-product of me finding redemption in code for the past two weekends, which is building on top of the projects of the past several weekends.
Also: I avoided all the macros for now, so no RipCurrent-like hell. And believe we can keep it this way for now. The user-facing code is just some lazy initialization; macros would make it bit cleaner to read the pipeline, and they could make the auto-generated dashboard a bit more informative, none of which sounds important to me so far.
Well, technically, I avoided all the macros except one: the one that wraps a pipeline, along with its configuration from FLAGS_*, into main(). But it's not a macro in the framework code but a macro in example code.
Last but not least: the directories, files, and class/method names would most certainly need to be changed. Historically, of us two, you are way better than me at this.
Hi @mzhurovich,
First of all, this is actually quite a simple piece. Especially if you first read the diff that is not automatically expanded by GitHub (i.e. everything but
dsl.handtest.cc), thentest.cc, and only then get to the verydsl.h.The main purpose of this code is to get rid of the error-prone manual specifications of
InputOf<>/OutputOf<>/SinkOf<>, which I used a while back to just get the test running. The secondary, yet important, purpose of this code is to support parallel execution (which is as simple as managing the "input available until / input processed until" trackers, as I wrote about here).The other, less important goals, are:
Ultimately, the within-machine node running its own pipeline would need a Karl-like status page. With a) tracking queue size (and what is stuck), b) tracking performance of each individual worker, and c) controls to change parameters on the fly, most notably, the processing block sizes per module (for the throughput / latency tradeoff).
These are all in some distant-ish future. For now, it's just the by-product of me finding redemption in code for the past two weekends, which is building on top of the projects of the past several weekends.
Also: I avoided all the macros for now, so no
RipCurrent-like hell. And believe we can keep it this way for now. The user-facing code is just some lazy initialization; macros would make it bit cleaner to read the pipeline, and they could make the auto-generated dashboard a bit more informative, none of which sounds important to me so far.Well, technically, I avoided all the macros except one: the one that wraps a pipeline, along with its configuration from
FLAGS_*, intomain(). But it's not a macro in the framework code but a macro in example code.Last but not least: the directories, files, and class/method names would most certainly need to be changed. Historically, of us two, you are way better than me at this.
Thanks, Dima