search

cloudfoundry / prometheus-boshrelease

bosh release for prometheus ecosystem
Apache License 2.0
112 stars 162 forks source link

Ability to monitor multiple kubernetes clusters on single BOSH director #294

Closed aegershman closed 5 years ago

aegershman commented 5 years ago

Is it possible to use this prometheus-boshrelease to monitor multiple kubernetes clusters managed by a single BOSH director?

background

We'd like to use Prometheus to monitor our kubernetes clusters, however I'm under the impression that the monitor-kubernetes.yml can be used for one deployment, however we have multiple clusters being managed by a single BOSH director. We'd prefer not to have a prometheus deployment for each cluster. Any clarification or advice is appreciated!

frodenas commented 5 years ago

You're right, the monitor-kubernetes.yml op-file can be used to monitor just a single kubernetes cluster.

If you want to monitor multiple clusters, you'll need to create your own op-file. The trick here is that you need to deploy multiple kube_state_metrics exporters, and as BOSH doesn't allow to deploy the same job multiple times in a instance group, you will need to deploy each exporter on a separate vm (maybe collocated with the k8s master node?).

Same happens with the scrappe config, but in this case, it's just adding multiple scrappe configurations.

aegershman commented 5 years ago

I realize this is an OSS project, so forgive me for bringing a vendored solution into this:

I'm using this with PKS (Pivotal Container Services), which deploys an OpsMgr for the BOSH director and the PKS cli when creating clusters. Does this change the context of the conversation at all? I think colocating the exporters on the k8s cluster wouldn't be possible if using the standard PKS CLI workflow.

Thoughts or feedback welcome (I might simply be up a creek for the time being)