[orchestration pipeline] Stop exporting OpenShift resources

[michaelsauter]

I have circled these thoughts with a few people already, but I realised I never wrote them down in a GitHub issue. So here it goes.

At the moment, the orchestration pipeline checks for each component whether an openshift folder exists. If it does, the pipeline assumes that this folder contains templates defining the OpenShift resources. If the folder does not exist, an export of all resources labelled with app=${projectId}-${componentId} is performed, and the resulting template is stored in openshift-exported. This feature was introduced to allow users to deploy their component without needing to think about / declare the resources in code. Instead, users can use the console UI to click their app together in the DEV environment, and then get this state replicated in QA/PROD. This approach poses a problem though: typically some parameters (e.g. DB access credentials) should be different between the environments, and to overcome this problem, there is a "reverse param" magic, which replaces hardcoded values in the exported template with parameter names, so that other values can be set for those parameters in QA/PROD.

Let's make this very clear - I think this whole idea of an "ad-hoc automated export" is a terrible mistake, for several reasons:

• Inferring the required resources from the current resources in DEV s error-prone: e.g. you might have created resources for experimental reasons only, or, you might have resources which you did not label correctly. To put it more strongly, if you do not define in source code what the resources should be, you are working with an undefined state.
• Using the UI to make changes to the resources kills traceability: you don't know who did something, you don't know when they did it, and you don't know for which reasons they did it. There is no link to any software design spec / functional requirement spec, and there is no responsible / author of the change. To my understanding, this is a no-go in a GxP context.
• The "reverse param" magic works only somewhat. There can be bugs (e.g. replacements happen that you did not want, as it is a crude search-and-replace operation), it's limited in that you can't make modifications in the template that stick around (every export overwrites the existing template), and most importantly, it is quite hard to explain to users (I had many questions about it). Having a feature that only works in the most basic case but doesn't work in a typical scenario is not worthwhile IMHO.
• Creating an export requires the tailor binary to be present. Unfortunately, the orchestration pipeline was designed to run on the Jenkins Master, but the tailor binary was only installed on the Jenkins agent in the beginning. Therefore, the orchestration pipeline does this "switch to the agent and back" dance for the export operation (note that now the Jenkins Master also has the tailor binary because there was a bug with one component type and this "dance" ... but having a binary in two places is a bad place to be and I'd rather not have this in the first place ...)
• The component pipeline now supports Helm next to Tailor, and the orchestration pipeline should support Helm too. Long-term, Helm will likely replace Tailor, as OpenShift has added tight integration with Helm since v4.3. Helm has a much more complex and powerful templating engine compared to OpenShift Templates (which is what Tailor uses). I have not found a tool yet that can properly export current resources to Helm templates. Even if there is such a tool (or we would build one), it would again suffer from the problem that no "template logic" can be conveyed in an export. A live export is an even worse fit for Helm templates.
• Creating an export in the orchestration pipeline requires us to commit and push the newly created/updated template. This means that during a pipeline run, we have a commit for which the pipeline is triggered, and we have a commit which is the result of the pipeline run. This is bad for several reasons:
  • It pollutes the repository with commits from a technical user
  • As each release version has its own branch, the commit ends up in one release branch, but not in the main branch (e.g. master). This can be confusing for users not too familiar with Git. As a consequence, the release branches are merged back into the main branch during an orchestration pipeline run. I deeply regret giving in to this feature request. I believe this feature has nothing to do with orchestration, is error prone (e.g. merge conflicts), and needlessly complicates the code. Instead, the orchestration should NOT create any commits. Consequently nothing needs to be merged back (note that user-made changes in a release branch should be done through pull requests, and pull requests can be merged automatically to the main branch using Bitbucket).

To sum it up, I strongly believe this feature should be removed. Using the auto-export is not something I would recommend to anyone, especially not in a GxP context. Removing the feature would simplify the documentation, it would simplify the code, it would allow us to easier integrate Helm and would allow us to undo other mistakes (like the merge-back) in future steps.

If we agree on removing it, we should however define how we want to provision new quickstarters. I believe the best would be to ship them with some templates so that there is no initial, manual effort. We'd need to pick if we want to ship with OpenShift templates or Helm templates (or allow the user to pick? or make that dependant on the OpenShift version?). I would love to take this opportunity to ship with Helm templates out-of-the-box, at least for OpenShift v4.

@clemensutschig @metmajer @martsec @segator @henrjk @gerardcl @oalyman Please provide feedback.

[segator]

Agree. From my point of view ODS should support multiple packager apps. as helm will work for basic applications but could not be enough for complex, at this point we'll need something like kustomize. I suggest start moving everyone to helm and avoid export. I Only see benefits. as final user will be aware of how their application is deployed and they will have control over deployment strategy and traceability. Even this open doors to reuse helm packages, decoupling build from deployment process on our jenkins jobs, (nexus have Helm repository support)

[michaelsauter]

Thanks @segator!

as helm will work for basic applications but could not be enough for complex, at this point we'll need something like kustomize.

kustomize does not allow me to do anything that I couldn't do with Helm to my knowledge. To the contrary, kustomize is a simpler approach than Helm. Further, until we have a real-world use case that cannot be covered with Helm, let's not complicate things. But let's keep this discussion separate as it is unrelated to the export functionality.

as final user will be aware of how their application is deployed and they will have control over deployment strategy

I am unsure about how much control should be given to each component. That is an area I need to understand more about. In a GxP context, I'm not sure how much customisation is allowed before the application author needs to account for these in their documentation. To my understanding, right now the application author simply defers to an existing validation of the pre-described deployment approach by the orchestration pipeline. That said, again this is a separate topic and not related to the export functionality so let's keep it out of the discussion for now.

[henrjk]

Please bear with me and let me put my spin on this to clarify the situation perhaps for others also:

• Infrastructure as Code. This is the definition of the infrastructure to checked into version control with the code (in the openshift folder). These definitions can be more abstract than openshift templates. For tailor the definitions are at the same level but tailor adds capabilities to manage parameters. As I have not used helm much I could be wrong but as there is go templating involved I would guess one can have definitions which are more abstract. In both case the infrastructure as code are essentially templates.

• Infrastructure Definitions. This is the output of applying actual parameters with tailor or helm to create infrastructure definitions. The outcome are openshift resource definitions in form of yaml files. Instantiation of templates destroys information as now things are no longer abstract and also it is no longer spelled out what pieces were parameters. However one can design instantiation so that it adds back information to enable traceability to the infrastructure as code version and potentially also the actual parameters used and perhaps more to enable repeatability. ODS currently captures build information (image labels) into the BuildConfig.

• Infrastructure State. An infrastructure state is represented by resources which go beyond resource definitions as it represents the actual state which also has current status information of the system as well as some history and deployment generation information to better reconcile updates. Note that this state can also be affected by using the openshift web console interactively or firing some oc commands.

To extract Infrastructure as Code from Infrastructure state is a losing proposition and I am fully supporting the idea to remove this from our code base.

[henrjk]

Regarding shipping quickstarters it appears that helm will not make sense for v3.11 so we should use tailor templates for that. I am also for using helm for OpenShift 4.

[albertpuente]

I was thinking today about this... Having helm charts included by default in quickstarters would simplify a lot understanding how to parametrize the deployment to different environments (as long as we carefully see how much we encourage to change). I do not see the automatic exporting of resources by the orchestration pipeline (nor doing it manually with tailor when starting working at a component level) as the most intuitive path for new users.

Has there been any decision been made on this regard? @michaelsauter

[michaelsauter]

To my knowledge, no.

I've added support for Helm in the component pipeline, but the orchestration pipeline does not handle Helm yet. It would need to handle Helm before we can ship quickstarters with Helm charts.

As mentioned above personally I'm all in favour for shipping Helm charts by default and not doing an export ... but I guess this is a bigger change where buy-in has to come from all sides, as a migration for existing users would be needed I think.

[metmajer]

@michaelsauter I have been discussing this topic a few days ago with @clemensutschig. Michael, can you outline what's needed to make the Release Manager with components that use Helm charts?

[michaelsauter]

Sure!

The component pipeline has a switch to either use Helm or Tailor, see https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/component/RolloutOpenShiftDeploymentStage.groovy#L126-L136.

The same would need to happen in https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/orchestration/phases/DeployOdsComponent.groovy#L51. This should be the main thing to do, but of course the devil is in the details :)

Right now I can think of the following places needing adjustment:

• exporting OpenShift templates for use with Tailor: this should probably not happen any longer when a Helm chart is present
• collecting "environment files": for Tailor there is some logic to automatically use param files if they are named using a certain pattern (e.g. foo-dev.env, foo-test.env, foo-prod.env etc.). A similar thing should be done for Helm.

Apart from this, there are a few differences between Helm and Tailor which would need some consideration. I'm just going to link to https://github.com/opendevstack/tailor/wiki/Migrating-from-Tailor-to-Helm, which offers some more details. One thing to call out here is that existing resources need to be explicitly adopted by Helm before they can be updated by Helm.

[serverhorror]

The component pipeline has a switch to either use Helm or Tailor, see https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/component/RolloutOpenShiftDeploymentStage.groovy#L126-L136.

For anyone looking, heres the ODS4 docs for this: https://www.opendevstack.org/ods-documentation/opendevstack/4.x/update-guides/4x.html#_helm

[serverhorror]

@michaelsauter We want to give a helm rollout a try.

From what I can see a good starting point is here:

• https://github.com/opendevstack/ods-jenkins-shared-library/blob/629fd7c1d97d168aa87ddb22a87be7e24e46bd22/src/org/ods/component/RolloutOpenShiftDeploymentStage.groovy#L126-L132

Additionally, the lines you mentioned later.

• collecting "environment files": for Tailor there is some logic to automatically use param files if they are named using a certain pattern (e.g. foo-dev.env, foo-test.env, foo-prod.env etc.). A similar thing should be done for Helm.

I'm thinking about a values-dev.yaml, values-test.yaml, values-prod.yaml but am not sure how we would like to handle a case with multiple charts, if we even support that by default or want to have a single chart that uses subcharts to get by with a single set of values-....yaml.

Do you prefer a specific issue or should we try and give the PR a reference to this issue?

[michaelsauter]

@serverhorror Have a look at https://github.com/opendevstack/ods-pipeline/blob/master/docs/tasks/ods-deploy-helm.adoc. In ODS Pipeline, we already have implemented a solution that works with multiple values files, and also allows to work with charts spread across repos. Maybe that helps or at least can serve as inspiration?

Do you prefer a specific issue or should we try and give the PR a reference to this issue?

Pick whatever you find suits best.