This PR adds support to get the element_processing_time metrics to beam.
We are actually calculating how much time it takes for an element to be processed during each stage, this information is passed with the element in the PCollection itself.
This is done in three parts:
AddTimer DoFn: Add this DoFn above the stage you want to calculate the metrics from.
timeit decorator: After the AddTimer stage, add this decorator to the process functions for all subsequent stages.
AddMetrics DoFn: Finally after the last stage, add this DoFn that adds the actual metrics.
This PR adds support to get the element_processing_time metrics to beam.
We are actually calculating how much time it takes for an element to be processed during each stage, this information is passed with the element in the PCollection itself.
This is done in three parts:
AddTimerDoFn: Add this DoFn above the stage you want to calculate the metrics from.timeitdecorator: After the AddTimer stage, add this decorator to theprocessfunctions for all subsequent stages.AddMetricsDoFn: Finally after the last stage, add this DoFn that adds the actual metrics.