jhellerstein commented 9 months ago

Code:

[ ] Associate hydroflow+ functions with hydroflow syntax for flamegraphs: https://github.com/hydro-project/hydroflow/issues/1268
[ ] #1269
[ ] #1270
- What kind of throughput are we getting
- (source or sink op id, tick, bytes, messages), time elapsed per tick

Tasks:

[ ] Figure out how perf flamegraphs work with idle time
[ ] Measure overhead from profiling

Need a way to get fine-grained telemetry and control over its aggregation/publishing within a transducer
Need ways to collect/aggregate telemetry across transducers
- write to S3/etc
- a library like Anna KVS with telemetry keys, sketch lattices, etc.
Telemetry standards and third-party tools
Beware of self-measurement abberations

While designing, think about what would make this a differentiating and general-purpose value-add of Hydro. Compare to Dapper etc for collection/aggregation.

MingweiSamuel commented 7 months ago

Tiemo:

How can I know when a batch of data is fully processed? (#834)
- Right now, have to implement a tombstone to signal EOS
- Have design patterns for termination detection
- (Have general-purpose termination detection as a std library? #346)

MingweiSamuel commented 3 months ago

834 related

rohitkulshreshtha commented 4 weeks ago

Next steps: Check with David re: metrics.

hydro-project / hydroflow

Telemetry: local and global #928

834 related