search

prometheus / client_python

Prometheus instrumentation library for Python applications
Apache License 2.0
3.96k stars 798 forks source link

OpenMetrics support for textfile collector #957

Open petarmaric opened 1 year ago

petarmaric commented 1 year ago

Currently prometheus_client.exposition.write_to_textfile doesn't support the OpenMetrics text format, which I'm addressing with this (private) hack:

from prometheus_client.openmetrics.exposition import (
    generate_latest as generate_latest_openmetrics,
)
from prometheus_client.registry import CollectorRegistry

def write_openmetrics_to_textfile(path: str, registry: CollectorRegistry) -> None:
    """
    Fork of `prometheus_client.exposition.write_to_textfile`, as the project
    didn't provide an OpenMetrics variant of this helper function.
    """
    with open(path, "wb") as fp:
        fp.write(generate_latest_openmetrics(registry))

Would your project be interested in a non-hacky (as is the case above) OpenMetrics support for textfile collector?

csmarchbanks commented 1 year ago

Hello, I am not opposed to adding a function to prometheus_client.openmetrics for textfile if it doesn't add confusion, but I am curious what your use case is? As far as I know the node exporter textfile collector only supports the Prometheus format, but I could have missed an update.

petarmaric commented 1 year ago

Thank you for the interest :) Honestly, can't give too many specifics right now, as I'm waiting for approval from the company I work for to actually Open Source the tool I'm building.

What I can say right now is that it's a console app and Python API for converting reports outputted by "a tool" into the OpenMetrics text format. Such metrics are then meant to be pushed into Grafana (data source), via VictoriaMetrics API.

petarmaric commented 1 year ago

Once it's Open Sourced I'll be sure to provide a link here, so you can see the use-case more clearly - there are a few reasons I've opted in for OpenMetrics in lieu of Prometheus text format...

petarmaric commented 11 months ago

As promised, https://pypi.org/project/locust_csv2openmetrics/ has (finally) hit 1.0.0 today.

We use this tool to convert locust’s load test CSV formatted reports into the OpenMetrics text format, pushing the obtained metrics into Grafana (data source), via VictoriaMetrics API.

petarmaric commented 11 months ago

Because these load tests are run non-continuously and at irregular time intervals we need to push their metrics into Grafana (data source), as opposed to the much more common way of Prometheus pulling data from a load-tests-like service to collect the metrics. I hope this clarifies our use case.

csmarchbanks commented 11 months ago

That use case makes sense to me, it is also the format you need to backfill data into Prometheus using promtool. That said, it looks like what you did is the correct thing to do, call generate_latest and save to a file. The reason write_to_textfile exists is that atomic writes are necessary in order to use the textfile exporter, but backfilling data does not require an atomic write.