search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

prometheus config-reload & localhost #76

Closed naude-r closed 4 years ago

naude-r commented 4 years ago

recently the client's network got a new dns entry, localhost.$client.com.

for some reason the config-reloader resolves localhost to above node, ie a lan ip and not 127.0.0.1

the pod's shell, using wget, does resolve localhost correctly, so not sure what the issue is.

would it be possible to allow this url to be configured?

Related operator issue and fix

carlosedp commented 4 years ago

Gonna wait until prometheus-operator and config-reloader gets a release to generate the new images and update the manifests. Thanks for reporting.

naude-r commented 4 years ago

did a quick test.

  1. modified manifests/setup/prometheus-operator-deployment.yaml
  2. add --localhost=127.0.0.1 to the prometheus-operator container
  3. deploy
  4. check reloader arguments: 
    cat /proc/1/cmdline 
/bin/prometheus-config-reloader--log-format=logfmt--reload-url=http://127.0.0.1:9090/-/reload--config-file=/etc/prometheus/config/prometheus.yaml.gz--config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml/

    from above we can see that your images are up to date. the linked proimetheus issue + PR was resolved Aug 28, 2018.

the next question. adding the argument manually to the manifest is not a big issue. is there a way to automate it though?

had a quick look and could not find an obvious way to do this.

any pointers?

carlosedp commented 4 years ago

It's tricky since it's in the prometheus-operator.libsonnet and it would need to be overridden.

naude-r commented 4 years ago

thank you. understood. in this case it is maybe easier to just manually modify the manifest.....this only happens during initial install after all.