revisit how we manage secrets and configuration

[mapellidario]

overview

In order to ease the operations, especially during during emergencies, we can streamline how we manage secrets and configurations around CRAB.

This is a broad issue and many smaller action items can be spawned to achieve this goal.

For example, references to REST secrets and config are found in the following places, while ideally only one is necessary.

• https://gitlab.cern.ch/cmsweb-k8s/services_config/-/tree/prod/crabserver , which should be the only source of truth.
• https://github.com/dmwm/CRABServer/blob/master/src/script/Deployment/Crabserver/CRABServerAuth.py
• https://github.com/dmwm/deployment/blob/master/crabserver/config.py
• https://github.com/dmwm/deployment/blob/c35738d451c4053cab6b55648ab2fdc44c327c7a/crabserver/deploy#L57

action items:

REST

• [ ] identify all the places where we are keeping some reference to the config
• [ ] deprecate all the ones that we do not need
• [ ] fix all the scripts that point to old config, if any
• [ ] think about if there is something better that we can do

TW, PUB

• [ ] (same as REST, i will expand this list while I progress)

past

We already have some open issues that are somewhat related to this one:

• https://github.com/dmwm/CRABServer/issues/5646
• https://github.com/dmwm/CRABServer/issues/6047

[novicecpp]

Copy from CRAB weekly minutes. Secrets management:

• Below are ideas circulated so far. We will have a discussions and hopefully converge after C&O week.
• sops+age, one key per environment, distributed keys via openstack barbican (manually) https://cms-http-group.docs.cern.ch/k8s_cluster/sops_age/
• Steps (per environment):
  • Create Age key
  • Use sops with Age key to encrypt CRABServerAuth.py
  • Commit CRABServerAuth.py and config.py to git and push gitlab
  • Store Age key inside openstack secret
  • Use decrypt-secrets.sh/encrypt-secrets.sh helper script to decrypt and re-encrypt the secret
  • Deploy Age key to kubernetes cluster
  • run usual deploy-secrets.sh to decrypt the CRABServerAuth.py and apply to kubernetes secrets as plain text
• Once this is done we should simplify things:
  • config.py is not a secret
  • secret file should only contain oracle password and S3 tokens
  • CRABServerAuth.py should be plain text without passwords
  • At pod start time passwords should be read from secret file
• Change to encrypt with gpg instead of age and use our own deploy-secrets.sh. Will update instructions soon.
  • https://gitlab.cern.ch/crab3/crab-secrets

[novicecpp]

Secret management so far:

• We changed secrets management from sops+age to sops+gpg, avoid sharing master key between us.
• We agreed to use https://gitlab.cern.ch/crab3/crab-secrets as source of truth, and to share configuration and credentials between us.
• We need regularly update files in https://gitlab.cern.ch/cmsweb-k8s/services_config/-/tree/prod/crabserver for CMSWEB team. They need it for deploy when upgrade cluster.
• Docs: https://cmscrab.docs.cern.ch/technical/crab-secrets.html

Dario, if you need any help to close this issue, feel free to ask.

[belforte]

this was done. @novicecpp do you agree to close ?

[belforte]

Maybe open a new one about how best to avoid forgetting to update secrets in Aroosha's repo when we update something on our side ?

[novicecpp]

Sorry, I do not remember why this still open.

In conclusion,

• All secrets must put in crab-secrets repo.
• File in crab-secret is encrypted with sops using gpg, which work quite well when we have a new guy join team.
• Copy the secrets manually to tbag, CI variables, or K8s Secret, etc.
  • This can be automated, not that hard, but take some time to do and might not worth effort where some secrets are rotated once a year. Anyhow we can start from https://github.com/dmwm/CRABServer/issues/8017

[belforte]

looks like we forgot to close. Thanks