Feature: Using in combination with a prometheus-operator

[FUSAKLA]

Hi, I was wondering if it would be possible to use the elector in combination with the prometheus-operator.

The change would be that instead of changing directly the config of Prometheus, the elector would modify the Prometheus CRD RemoteWriteSpec.

In the YAML it is

spec:
  remoteWrite:                                                                                                                                                                                                     
  - bearerTokenFile: /etc/prometheus/secrets/k8s.remote-write/token                                                                                                                                                
    queueConfig:                                                                                                                                                                                                   
      maxShards: 200                                                                                                                                                                                               
      minShards: 1                                                                                                                                                                                                 
    remoteTimeout: 12s                                                                                                                                                                                             
    url: https://foo.bar/k8s/push_internal/api/v1/push   

I assume that would require some work on the elector side, but just checking if that would even make sense for the project, and you would be open to possible contribution of this feature?

Thanks for considering it at all and sharing your work!

[FUSAKLA]

Looking at the code, that would lead to having two types of config reconciles. One for file and second using Kubernetes API and modifying the Prometheus CRD :thinking:

[jlevesy]

Hey @FUSAKLA, sorry for the delay, I missed your issue.

Thanks for reaching out. I'm open to contributions of course 😄.

That being said, I'm not familiar with the prometheus-operator at all, so it will take a little bit of time for me to catch up 😅 .

If it boils down to implementing another type of reconciler, I guess this is doable, but I wonder if it actually makes sense. Can you clarify how the prometheus-operator actually manages prometheus-agents instances? More specifically, do we have one spec per agent instance, or one shared spec with multiple replica?

[jlamillan]

An instance of a Prometheus Custom Resource(CR) is both the single source-of-truth for, and one-to-one mapping of the resulting StatefulSet and Prometheus configuration within the pods.

If the Prometheus configuration was changed by a prometheus-elector sidecar, it'd break the contract between the operator and resource spec. The operator would rightly revert any local changes to the configuration when it reconciled the configuration against the Prometheus CR.

If instead the prometheus-elector updated Prometheus.spec.remoteWrite of the custom resource directly as @FUSAKLA proposed, those changes would be applied to the configuration in all pods in that StatefulSet, which is not what we want since we presumably want a single leader and one or more followers.

Having separate Prometheus CR instances / StatefulSets for the leader and follower(s) gets us closer. However the Prometheus operator chart is set geared towards setting up a single Prometheus StatefulSet defined by a single Prometheus custom resource. Plus, theprometheus-elector seems designed to run as a sidecar rather than as an orchestrator of multiple instances (e.g. prometheus-elector -leader-custom-resource=prometheus-a -follower-custom-resource=prometheus-b -lease-name=...).

TL;DR it doesn't look like support for prometheus-operator is feasible without changing the design of the prometheus-elector works at a deeper level than the reconciliation logic.

[jlevesy]

Thanks for this detailed input @jlamilan!

If instead the prometheus-elector updated Prometheus.spec.remoteWrite of the custom resource directly as @FUSAKLA proposed, those changes would be applied to the configuration in all pods in that StatefulSet, which is not what we want since we presumably want a single leader and one or more followers.

I was fearing this TBH.

TL;DR it doesn't look like support for prometheus-operator is feasible without changing the design of the prometheus-elector works at a deeper level than the reconciliation logic.

How about extending the Prometheus CR (or actually defining a new one?) to make it declare a leader election? Then patch the operator reconciliation loop so that it provisions a statefulset that injects "prometheus-elector" as a sidecar? Would that make more sense?

We also have to account for the fact that the configuration file model is different for prometheus-elector, to allow defining the behavior in agent mode and in leader mode.

[yakir-levi]

Hi @jlevesy

First thank you for sharing this project!

we also use the Prometheus operator, in this case Prometheus operator automatically generates the configuration and store it in Secret that is mounted to Prometheus so we can't use the static configMap structure

follower:
  scrape_configs:
  - job_name:       'foobar'
    scrape_interval: 5s
    static_configs:
    - targets: ['localhost:8080']
    metric_relabel_configs:
      - action: labeldrop
        regex: "version"
  - job_name: "kubiznetes"
    scrape_interval: 10s
    kubernetes_sd_configs:
      - role: node

leader:
  remote_write:
    - url: http://remote.write.com

to handle this case we decided to do the following:

1. Add a flag that contains the leader config file (secret), so we have the Prometheus configuration file that is automatically generated by the Prometheus operator and updated every 1min and another file that contains the leader configuration

2. we read these 2 files contact and store them in the same config struct

type config struct {
    Follower map[string]any
    Leader   map[string]any

3. According to the pod's leader election, we apply the new configuration

I opened a PR with the changes https://github.com/jlevesy/prometheus-elector/pull/76

we have already deployed it to our production