Missing image tags when rollout to qa/test using helm

[quenteum]

Describe the bug We use a mono repo with multiple services inside. The build and the release to dev using the release-manager is working fine. When trying to release to qa/test we also had the bug https://github.com/opendevstack/ods-jenkins-shared-library/issues/1011. As i was verifying the bugfix https://github.com/opendevstack/ods-jenkins-shared-library/issues/1011 i have found another bug.

When releasing to qa/test only the image of the first deployment in the file chart/ods-deployments.json is imported:

Importing images - deployment: first-deployment, container: first-deployment, image: first-deployment@sha256:hash, source: source-ns oc -n namespace-test tag source-ns/first-deployment@sha256:hash first-deployment:ods-generated-v20230728-3-0b353-Q Tag first-deployment:ods-generated-v20230728-3-0b353-Q set to source-ns/first-deployment@sha256:hash. oc -n namespace-test tag first-deployment:ods-generated-v20230728-3-0b353-Q first-deployment:latest Tag first-deployment:latest set to first-deployment@sha256:hash.

After importing the image for the frist deployment the rollout using helm starts: DEBUG: Helm Config for first-deployment -> [type:helm, selector:app=namespace-app, chartDir:chart, helmReleaseName:app, helmEnvBasedValuesFiles:[values.env.yaml], helmValuesFiles:[values.yaml], helmValues:[registry:image-registry.openshift-image-registry.svc:5000, componentId:app], helmDefaultFlags:[--install, --atomic], helmAdditionalFlags:[]]

As helm does an atomic rollout all deployment are rolled out together. The rollout fails as there is only the image for the first one is present in the system. The problem can be found in the file src/org/ods/orchestration/phases/DeployOdsComponent.groovy.

At the moment there is a loop deploymentDescriptor.deployments.each { String deploymentName, Map deployment -> which imports the image for every deployment and then also does a deployment for the deploymentMean applyTemplates(openShiftDir, deploymentMean) which then causes the error. I'll provide a workaround as PR.

To Reproduce Steps to reproduce the behavior:

1. Use https://github.com/opendevstack/ods-jenkins-shared-library/pull/1023 as tmp ods version in jenkins to avoid the bug https://github.com/opendevstack/ods-jenkins-shared-library/issues/1011
2. Use a mono repo with more than one services
3. Deploy to dev using mro pipeline
4. Deploy to dev using mro pipeline
5. See error

Expected behavior All images should be imported before doing an atomic helm rollout. Separate the image import to qa/test from the deployment. Do the atomic helm deployment outside of the loop only once not for every deployment in chart/ods-deployments.json.

Affected version (please complete the following information):

• OpenShift: 3.12
• OpenDevStack 4.x

[quenteum]

I added a possible workaround: https://github.com/opendevstack/ods-jenkins-shared-library/pull/1026

This solves the problem as the images are imported before doing the rollout for each deployment. Then the deployment to qa/test works as expeted.

There is still the problem that the loop over all deployments does the helm rollout multiple times for every deployment in chart/ods-deployments.json. Only the first one has a real effect as it deploys everything.

I think it's not possible to change the structure of this json as it is used by many components but one idea i discussed with Martin Marcher would be to have only one DeploymentMean containing the deployments, so that we will do only on helm rollout containing all deployments.

[metmajer]

Hello @quenteum thanks for the fix, we're closing down some of our existing work and will look into this soon.

[hrcornejo]

Done in https://github.com/opendevstack/ods-jenkins-shared-library/pull/1026