search

SovereignCloudStack / moin-cluster-config

Configuration for our community cluster moin.k8s.scs.community
https://scs.community/
Apache License 2.0
2 stars 0 forks source link

Integrate moin cluster into scs monitoring #40

Closed michal-gubricky closed 4 months ago

michal-gubricky commented 4 months ago

This PR integrates moin cluster into our already existing SCS monitoring

Fixes #32 Related https://github.com/SovereignCloudStack/k8s-observability/pull/65

michal-gubricky commented 4 months ago

This PR does not yet contain Loki, only metrics for now.

mxmxchere commented 4 months ago

Hi michal, thanks for your nice PR, i would like to make a seperate PR for adding your pgp-keys, i propose the following approach:

  1. you make a new "keys"-PR with your key in hack/pgp_keys/mgubricky.pub.asc and your fingerprint in 75676F49EAFAA9CA .sops.yaml
  2. i reencrypt all secrets that currently exist and push the changes to the "keys"-PR, afterwards we merge
  3. we rebase the this PR and you can encrypt the secret with all keys, (i will figure out the command and send it to you when i have it :D) I suggest we add @chess-knight key in the same run, what do you think?

apart from that i like you PR, you picked up the existing structure quite nicely, that gives me the feeling that the folder structure gives new people a chance to get around.

michal-gubricky commented 4 months ago

Hi michal, thanks for your nice PR, i would like to make a seperate PR for adding your pgp-keys, i propose the following approach:

  1. you make a new "keys"-PR with your key in hack/pgp_keys/mgubricky.pub.asc and your fingerprint in 75676F49EAFAA9CA .sops.yaml
  2. i reencrypt all secrets that currently exist and push the changes to the "keys"-PR, afterwards we merge
  3. we rebase the this PR and you can encrypt the secret with all keys, (i will figure out the command and send it to you when i have it :D) I suggest we add @chess-knight key in the same run, what do you think?

apart from that i like you PR, you picked up the existing structure quite nicely, that gives me the feeling that the folder structure gives new people a chance to get around.

Hi Malte,

Sure, we can do that in a separate PR as well. I will create a new PR to add my key and, if @chess-knight wants, his key too. Yes, the folder structure is understandable, and as a new person, I had no problem understanding it.

mxmxchere commented 4 months ago

Looks good so far, in order to test the state i propose to renconcile the cluster on the state of this PR. We can then fix things that do not work well and merge a good state to main later.

In order to do that we have to set the reconciling target (https://github.com/SovereignCloudStack/moin-cluster-config/blob/e2e058b41e116a8903b7c414079da1ecfc0746a9/flux/config/flux-config-ks.yaml#L10) in this PR to the branch of this PR itself (otherwise the cluster will jump back to main).

The procedure to merge back to main when we are finished fixing will be the following

  1. suspend flux gitrepo resource
  2. set target (https://github.com/SovereignCloudStack/moin-cluster-config/blob/e2e058b41e116a8903b7c414079da1ecfc0746a9/flux/config/flux-config-ks.yaml#L10) in this PR to main
  3. merge
  4. set branch in flux gitrepo resource manually to main (via k edit)
  5. resume gitrepo
michal-gubricky commented 4 months ago

Looks good so far, in order to test the state i propose to renconcile the cluster on the state of this PR. We can then fix things that do not work well and merge a good state to main later.

In order to do that we have to set the reconciling target (

https://github.com/SovereignCloudStack/moin-cluster-config/blob/e2e058b41e116a8903b7c414079da1ecfc0746a9/flux/config/flux-config-ks.yaml#L10

) in this PR to the branch of this PR itself (otherwise the cluster will jump back to main). The procedure to merge back to main when we are finished fixing will be the following

  1. suspend flux gitrepo resource
  2. set target (https://github.com/SovereignCloudStack/moin-cluster-config/blob/e2e058b41e116a8903b7c414079da1ecfc0746a9/flux/config/flux-config-ks.yaml#L10 ) in this PR to main
  3. merge
  4. set branch in flux gitrepo resource manually to main (via k edit)
  5. resume gitrepo

It seems like the integration of the Moin cluster into the SCS monitoring is working. The Moin cluster can be seen at https://monitoring.scs.community/. Some manual steps need to be done on control plane components to expose metrics from them.

michal-gubricky commented 4 months ago

To expose the metrics of the control plane components, it was necessary to connect to each master node and edit the manifests of these components as described in the documentation. Please refer to the docs for details.

mxmxchere commented 4 months ago

manual changes to the controlplane components will get lost when a new node is created by CAPI. This will happen with the next update for example. But this is nothing we can fix in the context of this repo.

michal-gubricky commented 4 months ago

manual changes to the controlplane components will get lost when a new node is created by CAPI. This will happen with the next update for example. But this is nothing we can fix in the context of this repo.

Sure, I know that. These manual changes were the fastest way to test that monitoring is working correctly for the control plane components as well.

michal-gubricky commented 4 months ago

Logs from the moin-cluster should now be also available in SCS monitoring under a label moni-cluster.