Extend Spanner client metrics

[sebarthel89]

Is your feature request related to a problem? Please describe. We would like to receive more client metrics for spanner clients. Usually information like:

• retries
• latencies
• requests per second

allows to trace down issues by service-instance that uses spanner. There are server-side metrics that can not provide the same insights as metrics independent of a deployed service.

Describe the solution you'd like Right now, open census metrics are used to provide information about:

• oc_worker_spans
• sessions

It would be awesome if this list can be extended by

• retries
• latencies
• requests per second.

Right now, we use the Prometheus open census exporter to export the metrics that we can expose via Prometheus. In an ideal scenario, metrics would also be exposed using micrometer because it's part of the popular spring boot stack.

Describe alternatives you've considered

• building our custom metrics (Gauges, counters) for each service (already done in other teams, but this is a lot of work)
• building our own abstraction layer that would allow us to build a reusable spanner client that exposes metrics that can be reused
• not to do it and only rely on server metrics in combination with high-level client metrics like HTTP request latencies

Thanks in advance and keep up the good work!

[skuruppu]

Thanks for the feature request @sebarthel89. @asthamohta will take this on in a few weeks time when some other projects are wrapped up.

[olavloite]

@asthamohta @skuruppu I think it would make sense to pick up the feature request in #1268 together with this request, instead of separately add new metrics in different PRs.

[asthamohta]

Hi @sebarthel89 I started looking into your request and was wondering if you could share more about the metrics that you would like. When you say number of retries and requests per second do you mean number of retries for a transaction and number of transactions per minute? Similarly for latencies it'd be great if you could expand a little on what you expect.