Monitor Platform Credentials

[dduportal]

Service(s)

Other

Summary

Why

Quite often, we have processes failing because our platform credentials are expired. Should be smoother if we were proactive on these credentials.

What

Different class of credentials:

• Azure Service Principal (SP) Credentials used for API used for (non exhaustive):

  • Jenkins Core Release
  • Terraform/Packer infra. as code
  • Jenkins to Azure API to start agents (VM/containers)
  • Internal Azure service to service (AKS to storage for instance)

• AWS API accounts used for (non exhaustive):

  • Terraform/Packer infra. as code
  • Jenkins to Azure API to start agents (VM/containers)

• Other clouds (Digital Ocean, Oracle, Datadog, etc.)

• VPN certificates (CRL, individual accesses)

• jenkins-ci.org/jenkins.io certificates (Letencrypt)

• update center signing certificates

• release gpg keys

How

There are different monitoring, more or less complicated, but at least:

• Team shared google calendar (manual but efficient when it has event with notifications)
• Datadog monitoring probes

Other ideas?

Reproduction steps

No response

[timja]

You can also replace the Azure credentials with managed identities that Microsoft manages the credentials for, would only work for Azure agents though

[dduportal]

You can also replace the Azure credentials with managed identities that Microsoft manages the credentials for, would only work for Azure agents though

Interesting: that would remove a bit of the work. However I'm not sure how we could get the event" credential rotated" that would lead to a PR to puppet or helmfile/sops for us (to update the credentials for Jenkins controller), do you already have done such automated setup?

[timja]

you assign the identity to the VM or use a tool like https://github.com/Azure/aad-pod-identity when running on AKS.

you never see the credential directly, you just have a client ID and there's a service running on a non route-able IP that the access token is retrieved from (tools like AZ CLI, packer and terraform handle it mostly transparently, just need to set a flag)

Microsoft automatically rotates the credentials behind the scenes every 45 days but it's invisible.

[timja]

Another alternative to managed identity if support for other clouds is required is Workload federation https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation

Should be able to be done with: https://github.com/jenkinsci/oidc-provider-plugin

(although no sample for Azure, cc @jglick if interested)

[jglick]

I do not have an Azure account to play with. Note that oidc-provider is intended to supply temporary tokens as StringCredentials (or FileCredentials), mainly to builds though system scope is also possible, which may or may not be usable from the context of a Cloud or ArtifactManager or whatever. If you are talking about authenticating the whole Jenkins server to Azure, it probably suffices to use OIDC credentials supplied by infrastructure, like the service account associated with the pod running the Jenkins controller (if on K8s, else there are equivalents for EC2 etc.).