Integration of "system-metrics" configuration

[jochenehret]

Dear Community,

we would like to discuss the future integration of the system-metrics components into cf-deployment: https://github.com/cloudfoundry/system-metrics-release https://github.com/cloudfoundry/system-metrics-scraper-release

Currently the configuration is defined in an experimental ops-file: https://github.com/cloudfoundry/cf-deployment/blob/main/operations/experimental/add-system-metrics-agent.yml

As the system-metrics components are mature, we could either promote the experimental ops file to a regular ops file (=opt-in) or integrate the configuration into cf-deployment.yml with a new ops file to disable (=opt-out).

To make a good decision, we would like to know from cf-deployment consumers

• if they use the system-metrics components
• and if yes, which integration option they prefer

Please add a short comment to this issue. Thanks for your collaboration!

Jochen.

[lodener]

For a better insight into VM health, we make use of the node-exporter-boshrelease. This provides us with a wider array of metrics when compared to the system-metrics component. An explicit opt-in to the system-metrics component would therefor have our vote.

[chombium]

Hi,

at SAP, we are using a custom, home-grown monitoring solution for the system metrics based on Telegraf, so for us the promotion of the system-metrics-release and the system-metrics-sraper-release is irrelevant. Whatever integration method is chosen we will (leave) deactivate(d) the releases via an custom ops file.

I was analyzing the code of the both releases and "played a bit" with them and I can write few pros and cons for the releases:

• Pros:

  • If the releases are activated, on gets system metrics monitoring out-of-the-box
  • Integrated system metrics in Loggregator
  • If someone uses other BOSH releases apart from the defined in the cf-deployment, the same method can be used for collection of system metrics from the other BOSH deployments deployed on the same BOSH director. The system-metrics-release must be deployed on all VMs where the metrics scraping is needed. The scraping configuration is read by default from /var/vcap/instance/dns/records.json which is updated regularly by the bosh-dns.

• Cons:

  • The metrics go through Loggregator, so in case one uses the Loggregator's Firehose Architecture if the Doppler and RLP nodes get high load, they might start dropping log envelopes (logs and metrics) and the delivery of the system metrics might be unreliable. This can be mitigated by scaling properly the Doppler and RLP nodes and setting application log rate limits. One should never let some application get wild and log to many logs, because the noisy application might cause log drops in the Dopplers or RLP nodes.

[chombium]

For a better insight into VM health, we make use of the node-exporter-boshrelease.

@lodener I see that the latest release of this bosh release was done in 2020 and it uses the Prometheus node-exporter v1.0.1. I'd suggest that you update the release as the latest release of the node-exporter is 1.5.0 ;)

[Benjamintf1]

I still think there's a strong benefit to accessibility and consistency within cf with regards to metrics. In that respect, I think because there's some benefit with system-metrics release. I think there's also benefits to be able to edit and maintain metrics with respect to how they're useful for cf. That said, it wouldn't be that hard to tie node-exporter into a vision that includes the consistency and accessibility I think is important.

All that said, I'd be ok just promoting system-metrics and the scraper to being a regular opsfile rather then an experimental one.