Monitoring pipelines from the outside

[NiccoloOlivieriAchille]

Is your feature request related to a problem? Please describe. In our company we use centralised systems to monitor "things" (infrastructure, applications, containers, lambdas, ...) and it would be nice to have the possibility to monitor mage's pipelines from that centralised system, without having to open mage (which also means less people having the authorisation to access it)

Describe the solution you'd like I'd like to have "exports" of what's happening inside mage in near real time

Describe alternatives you've considered Right now I see to "easy" possibilities:

• have dedicated logs for high level pipelines, with precise messages to show the execution and problems of pipelines
• have another opentelemetry service which only shows pipelines, with timings, logs, results and so on...

[MartinLoeper]

I second this idea! I am looking for a solution which involves writing logs to fluentd to store them in Grafana Loki.

Also, it would be great to export Pipeline-Level logs in a Prometheus compatible format. I read about a metric endpoint being already implemented, but I think pipeline runs are not included as metrics, are they?