Cerberus monitoring - Githubissues

BCDevOps / platform-services

Collection of platform related tools and configurations

Apache License 2.0

13 stars 29 forks source link

Cerberus monitoring #752

Closed ShellyXueHan closed 3 years ago

ShellyXueHan commented 3 years ago

Background:

Idea: https://app.zenhub.com/workspaces/platform-experience-5bb7c5ab4b5806bc2beb9d15/issues/bcdevops/developer-experience/828

Task: https://app.zenhub.com/workspaces/platform-experience-5bb7c5ab4b5806bc2beb9d15/issues/bcdevops/developer-experience/1040

Changes:

test setup cerberus in klab cluster
setup uptime bot
devops manifest for building and deploying
remove need for privilege access: https://app.zenhub.com/workspaces/platform-experience-5bb7c5ab4b5806bc2beb9d15/issues/bcdevops/developer-experience/1078

Notes:

for next sprint I'll work on CCM for Cerberus and test install on silver
also need to look into custom checks for better monitoring alerts

gnunn1 commented 3 years ago

Is there a reason for building the image instead of just using the pre-built and maintained image at quay.io/openshift-scale/cerberus?

ShellyXueHan commented 3 years ago

@gnunn1 yea two reasons:

to get rid of the use for privileged filesystem access and the service account: https://github.com/BCDevOps/platform-services/blob/cerberus-monitoring/monitoring/cerberus/Dockerfile#L21 It's only using the privileged access to read/write output file, so it should not be necessary.
i also want to update the use for latest image tag from dockerfile

gnunn1 commented 3 years ago

Thanks for the explanation

StevenBarre commented 3 years ago

Is there a reason to git clone the latest instead of just installing via pip? https://github.com/cloud-bulldozer/cerberus/blob/master/docs/installation.md#python-package

ShellyXueHan commented 3 years ago

good points @StevenBarre

I don't know the specific reasons for the dockerfile setup, it's a copy from Cerberus repo and I haven't checked any details since it's just a POC. But before this make it to CCM I'll look into your suggestions and update them.

I'd also prefer to build and deploy Cerberus in its own namespace, but might have to setup the service account rolebinding in openshift-monitoring still (TBD, need to double check)

StevenBarre commented 3 years ago

but might have to setup the service account rolebinding in openshift-monitoring still

To get access to Prom you just need a CRB with get on namespaces. https://github.com/bcgov-c/platform-tools/blob/ocp4-base/ocp4/nagios/nagios.yaml#L26-L31

ShellyXueHan commented 3 years ago

@StevenBarre I've updated the devops manifests and tested in klab. Once those are good, I'll start to look into CCM and work on some playbooks to generate updated token and config from templates instead of commands.