Open stephen-soltesz opened 6 years ago
etl_task_count
- should be labeled with the target table sooner. "unknown" is difficult to work with in the dashboard.etl_worker_duration_seconds
- should use separate worker targets for separate file types -- this moves the decision to the queue pusher or much earlier in the request handling.etl_panic_count
- should indicate which parser is generating panics.
One day, a single parser service will handle tasks for multiple data types. When that occurs, we want to be able to create a dashboard that distinguishes the behavior for each data type.
This began by adding new "table" labels to several common metrics: https://github.com/m-lab/etl/pull/415
Then generalizing the usage of FileSizeHistogram and RowSizeHistogram to apply to all parsers. https://github.com/m-lab/etl/pull/443 https://github.com/m-lab/etl/pull/448 https://github.com/m-lab/etl/pull/450
There are a few metrics remaining that should be distinguishable.
This work also supports efforts to unify the parser dashboards for monitoring.