Export prometheus metrics

[tomershafir]

Export prom metrics to monitor the agent. I think Prometheus protocol is suitable and the otel superset is an overkill. Need to evaluate prometheus/client_java vs dropwizard/metrics.

I can probably contribute this, let me know if relevant

[korniltsev]

What type of metrics would you like to export?

We would like to have as little dependencies as possible, probably even getting the rid of okhttp and co.

Do you think we can do this pluggable / optional ?

[tomershafir]

I think it useful to have metrics visibility in prod on profiler overhead and valid operation, for start maybe: /health endpoint, error rates, upload latency, meminfo exposed by ap, transmitted bytes, build info, config info. Further stack unwind latency and USE (Brendan Greg) metrics can be considered later.

Regrading socket and service discovery, I guess we can try select a port reasonably and make it configurable.

However, this may overload scrapers and time series db. Maybe granular config like meta, errors and traffic, to let the client control amap.

wdym by pluggable?

[korniltsev]

By pluggable I meant it should be optional. Users who dont want metrics should not have metrics dependencies (clientjava or others). Maybe abstract it behind an interface or expose java APIs in pyroscope-java allowing to export metrics outside pyroscope-java.

[tomershafir]

Following curr config flow, the way is configurable dynamic class loading behind an interface. I understand you want minimum deps for security and maintenance, but dont you think its an essential one? I think the common case is users would need visibility.

[tomershafir]

https://github.com/prometheus/prometheus/issues/3756 this is a problem, I see 3 options:

1. prom/kube will fix it
2. sidecar/background merge manager that multiplex several ports into one (external to the agent)
3. shared registry e.g. if started from main()

we can prefer 3 and support the 2 above libraries, and fall back to a distinct endpoint to be handled externally