gitlab builds fail to start

[amnonkhen]

gitlab runner stopped working and fails to restart. It means our CI/CD is stuck, and no releases can be made.

This is similar to ebi-ait/dcp-ingest-central#958.

gitlab was upgraded from 15 to 16. last runner version was 0.52.1 which matches gitlab 15.11.1 gitlab is now 16.9.3 (see in https://gitlab.ebi.ac.uk/help) which needs runner version 0.62.1.

The tasks:

• [x] upgrade runner in ingest-kube-deployment/infra/helm-charts/gitlab-runner/Chart.yaml
  • [x] There might be other changes needed in the current runner setup due to deprecations in the transition from gitlab 15 to 16
• [ ] find out in gitlab slack channel how gitlab upgrades are advertised and discover these things earlier.

[amnonkhen]

The procedure that fixed the issue:

• create a new runner in the gitlab ui and get its new token
• update place holder of secret runner-token in /ingest-kube-deployment/apps/secrets/templates/deployment.yaml
• update secret value in aws secret manager
• redeploy secrets to k8s
• redeploy runner to k8s

We also updated the runner's readme file with this inforamtion.