[TT-1755] Feature request: Expose prometheus metrics endpoint

[ryanwalls]

Do you want to request a feature or report a bug? Feature

Would be great if a prometheus metrics endpoint could be exposed so we could scrape metrics about API activity.

[buger]

Actually, I think this feature should be implemented on the Tyk Pump side as a separate pump.

[letzya]

Another client's request. At the moment they have statsD between Prometheus and Tyk but would prefer to have it OOTB.

[dijitali]

This sounds great for the Tyk Analytics data and I think the idea of using the Tyk Pump to Prometheus was mentioned here: https://community.tyk.io/t/monitoring-tyk-pump-with-prometheus/1856

However for non-analytics data (like those currently output in Statsd or New Relic format) it would be seem more logical for them to be available in Prometheus format from that Tyk service directly as an additional instrumentation output type?

A workaround at the moment would be to use the StatsD to Prometheus metrics exporter but this seems a bit convoluted.

[asoorm]

https://github.com/prometheus/statsd_exporter is the most sensible solution IMO.

Prometheus gets metrics via pull, and does not support push. Meaning Prometheus will need to know the endpoint of each node in the cluster (whether that be gateway/pump or any other microservice).

Furthermore, via the pull model, the gateways will need to be able to prepare the data for prometheus when it queries the API. In terms of performance, the gateways should be spending CPU cycles for processing requests, not for log aggregation.

All microservices can push via udp the analytics to the statsd exporter - and now, Prometheus only needs to know and scrape from the endpoint of the exporter(s), rather than every service. Nicely de-coupled.

[dijitali]

Prometheus identifies current gateway/pump/dashboard pods via kubernetes label selectors in the service discovery config.

The statsd_exporter documentation seems to indicate that, although it is possible, it's not recommended to be used as a direct transport from StatsD to Prometheus. Since Tyk already supports two other metric outputs (one of which is a closed-source, commercial tool), Prometheus support doesn't seem illogical. Especially given that it's a suggested best practice for an application operating in Kubernetes:

Make your application easy to monitor Like logging, monitoring is an integral part of application management. A popular option in the Kubernetes community for monitoring is Prometheus, a system that can automatically discover the pods and services it has to monitor.

Prometheus support seems to be a common feature for most applications that operate in a Kubernetes environment: Ambassador and Kong API Gateways seem of offer this sort of functionality natively or via a plugin.

With regards to performance: there will be a CPU/memory penalty from any sort of metrics functionality, Prometheus, StatsD or otherwise. However it shouldn't have an impact unless I choose to use that type of monitoring?

I appreciate Prometheus support is mainly relevant for Kubernetes-hosted implementations and that may not be the current aim of this project. I'm just suggesting this as feedback for how the Tyk product might be improved.

[Bouwdie]

What is the status of this ticket? We would like all of the software that we use to provide us with sufficient instrumentation to monitor and act on the infrastructure level. If we have some way to look at the internals of the gateway we can bring it under control more efficient. We instument our go applications with https://prometheus.io/docs/guides/go-application/ Out of the box this client already provides us with valuable instrumentation like number of go-routines and memory usage.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, please add comments to this ticket if you would like it to stay open. Thank you for your contributions.

[excieve]

Prometheus is pretty much a standard with k8s-based monitoring. With Prometheus k8s operator it's very easy to manage service discovery through CRDs. So I'd say this should still be considered at some point.

[letzya]

We have added prometheus pump. https://github.com/TykTechnologies/tyk-pump/#prometheus

[letzya]

Reopening since this issue is for the gateway

[eloo]

Any updates here? We would really like to have an option to monitor the tyk-gateway itself.

[letzya]

@eloo FYI statsd-exporter is used by others https://github.com/TykTechnologies/tyk/issues/979#issuecomment-353557028

[SonjaChevre]

hi everyone!

here is an update from our side: in terms of observability we are currently looking at unifying our stack using OpenTelemetry, see https://community.tyk.io/t/faq-opentelemetry-distributed-tracing/5682.

OpenTelemetry is already supported by a large number of observability vendors: https://opentelemetry.io/ecosystem/vendors/. Should you use case really require the Prometheus format to export the basic health metrics (request rates, error rates, duration), you could leverage the OpenTelemetry exporter with the Prometheus exporter: https://opentelemetry.io/docs/reference/specification/metrics/sdk_exporters/prometheus/.

Would this solve all your use cases? Let me know - this is a topic we are actively working on.

Sonja

[SonjaChevre]

Here's a demo using OTel traces to expose Prometheus metrics for the APIs: https://github.com/TykTechnologies/demo-api-observability-opentelemetry

Please provide feedback directly on the community post: https://community.tyk.io/t/api-observability-dashboard-with-opentelemetry-and-grafana/6580/2 or by opening a new one.

We are planning to explore OTel metrics in a next step (exporting OTel metrics directly from Tyk GW).