collect critical metrics

yuanbohan commented 7 months ago

Purpose

help users to gain insight into the sdk behavior

How To

use Prometheus to instrument this SDK
export /metrics endpoint
this behavior SHOULD BE optional, and related setting SHOULD BE placed in Config, like: gorm prometheus

Metrics

request duration histogram
request count
error count
Go gRPC related metrics
other helpful metrics

Reference

tisonkun commented 6 months ago

May you try to add some descriptions and related code?

JetSquirrel commented 5 months ago

I'd like to do this issue.

tisonkun commented 5 months ago

@JetSquirrel The issue description is updated with more information. Assigned and please go ahead :D

JetSquirrel commented 5 months ago

@yuanbohan I instrument some metrics, do you think this metrics is enugh or we need add more? And how about the format of metrics , it's as expected or need to modify?

$ curl localhost:4010/metrics

# HELP greptime_client_go_request_count 
# TYPE greptime_client_go_request_count counter
greptime_client_go_request_count 3
# HELP greptime_client_go_request_duration_histogram 
# TYPE greptime_client_go_request_duration_histogram histogram
greptime_client_go_request_duration_histogram_bucket{le="0.001"} 0
greptime_client_go_request_duration_histogram_bucket{le="0.01"} 3
greptime_client_go_request_duration_histogram_bucket{le="0.1"} 3
greptime_client_go_request_duration_histogram_bucket{le="0.3"} 3
greptime_client_go_request_duration_histogram_bucket{le="0.6"} 3
greptime_client_go_request_duration_histogram_bucket{le="1"} 3
greptime_client_go_request_duration_histogram_bucket{le="3"} 3
greptime_client_go_request_duration_histogram_bucket{le="6"} 3
greptime_client_go_request_duration_histogram_bucket{le="10"} 3
greptime_client_go_request_duration_histogram_bucket{le="20"} 3
greptime_client_go_request_duration_histogram_bucket{le="60"} 3
greptime_client_go_request_duration_histogram_bucket{le="+Inf"} 3
greptime_client_go_request_duration_histogram_sum 0.006428089
greptime_client_go_request_duration_histogram_count 3
# HELP grpc_client_handled_total Total number of RPCs completed by the client, regardless of success or failure.
# TYPE grpc_client_handled_total counter
grpc_client_handled_total{grpc_code="OK",grpc_method="Handle",grpc_service="greptime.v1.GreptimeDatabase",grpc_type="unary"} 3
# HELP grpc_client_msg_received_total Total number of RPC stream messages received by the client.
# TYPE grpc_client_msg_received_total counter
grpc_client_msg_received_total{grpc_method="Handle",grpc_service="greptime.v1.GreptimeDatabase",grpc_type="unary"} 3
# HELP grpc_client_msg_sent_total Total number of gRPC stream messages sent by the client.
# TYPE grpc_client_msg_sent_total counter
grpc_client_msg_sent_total{grpc_method="Handle",grpc_service="greptime.v1.GreptimeDatabase",grpc_type="unary"} 3
# HELP grpc_client_started_total Total number of RPCs started on the client.
# TYPE grpc_client_started_total counter
grpc_client_started_total{grpc_method="Handle",grpc_service="greptime.v1.GreptimeDatabase",grpc_type="unary"} 3

yuanbohan commented 4 months ago

@JetSquirrel Thanks a lot for your time and commitment.

These metrics are enough for the first version. But We can discuss the following concerns:

all metrics names SHOULD BE started with a same prefix, like: greptimedb_ingester_go_ (or a better option)
unary or stream SHOULD BE a label in metric
greptime_client_go_request_duration_bucket (which is a Prometheus histogram, and I omited the histogram in the metric name) and greptime_client_go_request_duration_count contains the totol request. Is greptime_client_go_request_count needed?
what's the actual meaning of msg in _msg_sent_total and _msg_received_total?
what's the purpose of _started_total and _handled_total?

By the way, we should:

an option in config to control whether to export the /metrics endpoint or not (users may have already instrumented their apps already, and do not want sdk to export a port)
a Go function to get the metrics (users may have already instrumented their apps already, and want to merge sdk metrics into their own)

To save your time, we can discuss the metrics pattern before you implement them. 👍

yuanbohan commented 4 months ago

@JetSquirrel By the way, if you have time to acquire more practical and pragmatic patterns from the following links, it may help a better solution:

GreptimeTeam / greptimedb-ingester-go