search

cloudfoundry / prometheus-boshrelease

bosh release for prometheus ecosystem
Apache License 2.0
113 stars 163 forks source link

bosh and cf exporter getting errors and causing bosh director CPU to be overconsumed #386

Closed roberto-yslas closed 1 year ago

roberto-yslas commented 4 years ago

I am using this deployment with minor modifications for three different environments, two are working and one is having issues between the above mentioned exporters and bosh. I am colocating firehose exporter on the prometheus2 VM. Those are the extra ops files. I am deploying this in a bosh director not associated with the PCF PCF platform being monitored.
I have found other similar issues in here but no specific resolutions.


  --var-file bosh_ca_cert=${PVT_PROMETHEUS_BOSH_DEPLOYMENT_REPO}/lwr/root-ca-cert \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/monitor-bosh.yml \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/enable-bosh-uaa.yml \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/configure-bosh-exporter-uaa-client-id.yml \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/monitor-cf.yml \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/local-cf_exporter.yml \
  -o ${PROMETHEUS_BOSH_DEPLOYMENT_REPO}/manifests/operators/colocate-firehose_exporter.yml \
  -v deployment_name=${DEPLOYMENT} \
  -v uaa_bosh_exporter_client_id=bosh_exporter \
  -v bosh_url=10.195.166.10     \
  -v metrics_environment=lwr \
  -v metron_deployment_name=cf-ae8adbb1bbcd8905494f \
  -v system_domain=sys.pivotal-cloud-foundry.com \
  -v traffic_controller_external_port=443 \
  -v network=snet-gfpte-pvt-control-nane \
  -v azs=northamerica-northeast1-c \
  -v vm_type=large.disk \
  -v warning_to='user.name@customer.ca' \
  -v warning_from='dev-warning@customer.ca' \
  -v critical_to='user.name@customer.ca' \
  -v critical_from='dev-critical@customer.ca' \
  -v skip_ssl_verify=true --no-redact```

I extracted the following from the prometheus2 vm bosh and cf exporter logs:

```prometheus2/850a7a24-f6be-438b-9405-557d195239fd:/var/vcap/sys/log/bosh_exporter# cat bosh_exporter.stderr.log
time="2020-04-02T20:23:29Z" level=info msg="Starting bosh_exporter (version=3.3.0, branch=master, revision=63f29d319ef420bd0f6c85e5009b0f8ce5db8850)" source="bosh_exporter.go:277"
time="2020-04-02T20:23:29Z" level=info msg="Build context (go=go1.12.8, user=root@193b22c7c296, date=20190830-21:27:43)" source="bosh_exporter.go:278"
time="2020-04-02T20:23:29Z" level=info msg="Using BOSH Director `p-bosh` (94ed3490-c669-4f81-930a-bd273f3f53f7)" source="bosh_exporter.go:291"
time="2020-04-02T20:23:29Z" level=info msg="Listening on :9190" source="bosh_exporter.go:365"
2020/04/02 20:25:26 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:27:28 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:29:28 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:31:37 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:33:33 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:35:29 http: superfluous response.WriteHeader call from github.com/bosh-prometheus/bosh_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)```

```prometheus2/850a7a24-f6be-438b-9405-557d195239fd:/var/vcap/sys/log/cf_exporter# cat cf_exporter.stderr.log
time="2020-04-02T20:23:30Z" level=info msg="Starting cf_exporter (version=0.10.0, branch=master, revision=9ad8b3c7b5b33e663c31f4bdf4cab5291b60ccb7)" source="cf_exporter.go:136"
time="2020-04-02T20:23:30Z" level=info msg="Build context (go=go1.12.10, user=root@c5a0d42674df, date=20191015-20:22:59)" source="cf_exporter.go:137"
time="2020-04-02T20:23:31Z" level=info msg="Listening on :9193" source="cf_exporter.go:245"
2020/04/02 20:24:20 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:25:10 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:26:09 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)
2020/04/02 20:27:00 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:58)```
benjaminguttmann-avtq commented 1 year ago

Closing due to inactivity; Fell free to re-open if issue persists;