search

prometheus-operator / prometheus-operator

Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes
https://prometheus-operator.dev
Apache License 2.0
9.16k stars 3.73k forks source link

Allow users to use OTEL Kubernetes semantic conventions #6432

Open simonpasquier opened 8 months ago

simonpasquier commented 8 months ago

Component(s)

Prometheus, PrometheusAgent, ServiceMonitor, PodMonitor, Probe

What is missing? Please describe.

Right now the operator hardcodes the target labels in the scrape configurations:

OpenTelemetry uses different conventions for Kubernetes resource attributes (https://opentelemetry.io/docs/specs/semconv/attributes-registry/k8s/) which aren't possible to apply right now but once Prometheus supports UTF-8 characters for metric and label names, the operator should offer a way to opt-in for the OTEL conventions.

Describe alternatives you've considered.

No response

Environment Information.

Environment

N/A

haanhvu commented 8 months ago

So we need to wait for https://github.com/prometheus/prometheus/issues/13095 to finish right?

simonpasquier commented 8 months ago

@haanhvu yes

jmichalek132 commented 7 months ago

Hi, would it make sense to implement support in the mean time for the naming of the resource attributes with dots being replaced with underscores? If so I would be willing work on that.

simonpasquier commented 7 months ago

@jmichalek132 yes it sounds like a good approach to me.

@ArthurSens @nicolastakashi @slashpai more thoughts on this?

ArthurSens commented 6 months ago

Changing behavior after we implement the API (using dots once Prometheus supports it) doesn't sound like a good approach, so I would say yes if we put this behind a feature flag.

From my understanding, it is still unclear within the Prometheus team itself how UTF-8 will be supported. It wouldn't make sense for us to commit to a stable feature until Prometheus does it.

haanhvu commented 6 months ago

Hi, would it make sense to implement support in the mean time for the naming of the resource attributes with dots being replaced with underscores? If so I would be willing work on that.

Seems like this is not finalized yet? It's one of the things being asked in user survey: https://x.com/opentelemetry/status/1785657674861809910

simonpasquier commented 6 months ago

I would say yes if we put this behind a feature flag.

Agreed, I would definitely marketed it as experimental/subject to change.

simonpasquier commented 6 months ago

Thinking more about it, we can also allow users to fully customize the target labels in which case OTEL conventions become a special case.

Completely made-up API:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  ...
spec:
  targetMetadataMapping:
    pod: "k8s_pod_name" # it can be changed to "k8s.pod.name" once/if supported by Prometheus
    namespace: "k8s_namespace_name"
    container: "" # use the default "container" label
jmichalek132 commented 3 months ago

Hi, would it make sense to implement support in the mean time for the naming of the resource attributes with dots being replaced with underscores? If so I would be willing work on that.

Seems like this is not finalized yet? It's one of the things being asked in user survey: https://x.com/opentelemetry/status/1785657674861809910

Blog post discussing the results of the survey https://opentelemetry.io/blog/2024/prometheus-compatibility-survey/