Request to add Alerts examples for Client side metrics

[MosheBlumbergX]

As a customer I would like to see additional examples for client metric alerts and thresholds.

Very much like what has been outlined here:

https://github.com/confluentinc/jmx-monitoring-stacks/tree/main/jmxexporter-prometheus-grafana/assets/prometheus/prometheus-alerts

Thanks, Moshe.

[hifly81]

@MosheBlumbergX I was wondering what metrics should be alerted.

For producers: kafka.producer:type=producer-metrics,client-id="{client-id}" (record-error-rate)

For consumers: kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}" (records-lag-max)

kafka.consumer:type=consumer-coordinator-metrics,client-id="{client-id}" (sync-rate)

any other ideas?