component clashes with existing resources during migration

henrjk commented 2 years ago

I am exploring to adopt ods-pipeline in an existing project, so I'd like not to disturb the existing services.

An existing OpenShift resources of the service clashes with the components helm chart's service. While aiming to change the component name as used in the ods-pipeline to <original-name>-helm I ended up editing the component parameter in the TriggerBinding ods-pipeline, with a wild guess of setting it to value: $(body.extensions.repository)-helm

Perhaps there is an easy way to customize this which I haven't found yet. Perhaps this could be or should be declarable in ods.yaml. An alternative could be to change the resources in the templates chart. However then the image streams could still interfere.

michaelsauter commented 2 years ago

Did you try to set the release-name parameter in https://github.com/opendevstack/ods-pipeline/blob/master/docs/tasks/ods-deploy-helm.adoc#parameters? You'd also need to take care of the image stream, I think that can be done via https://github.com/opendevstack/ods-pipeline/blob/master/docs/tasks/ods-package-image.adoc#parameters ...

michaelsauter commented 2 years ago

Does this work now? Do we need to document this better?

henrjk commented 2 years ago

@michaelsauter I have not yet played with release names, but I tried some other strategies as follows.

For all of the following let's assume the bitbucket project is called hoho .

One thing that stopped me investigating the release (for no really good reason) was that:

  - name: deploy-helm
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm-v0-1-1
      release: hoho{{params.pr-key}}

did not work and I could not figure out quickly what syntax to use and what template capabilities are available in this context.

Instead I experimented with the following approaches to keep the helm managed resources separate.

Use a different name in `Chart.yaml` from what the component is associated with in ods Jenkins.

For repo hoho-auth I changed the chart name to dasher and adjusted the image-stream parameter as follows to make it work:

 - name: package-image
    taskRef:
      kind: ClusterTask
      name: ods-package-image-v0-1-1
    params:
      - name: image-stream
        value: auth-dasher

Use `fullnameOverride`

For repo hoho-admin I changed fullnameOverride to admin-helm chart name to dasher and adjusted the image-stream parameter to:

 - name: package-image
...
    params:
      - name: image-stream
        value: admin-helm

Use `nameOverride`

For repo hoho-backend I changed nameOverride to backend-helm. There was no need to adjust the image-stream parameter.

michaelsauter commented 2 years ago

@henrjk Ah interesting!

Here's the syntax for params (see also https://tekton.dev/docs/pipelines/pipelines/#specifying-parameters):

- name: deploy-helm
  taskRef:
    kind: ClusterTask
    name: ods-deploy-helm-v0-1-1
  params:
    - name: release-name
      value: hoho$(params.pr-key)

henrjk commented 2 years ago

I have been experimenting with the release-name parameter. The scenario I am after is less to avoid clashes with existing Jenkins/tailor managed resources, but to enable preview deployments.

The following kind of works but requires to also change the image-stream parameter like so:

...
  - name: package-image
    taskRef:
      kind: ClusterTask
      name: ods-package-image-v0-2-0
    params:
      - name: image-stream
        value: cool-admin-feature
    runAfter:
    - build-fe
    - build-be
    workspaces:
    - name: source
      workspace: shared-workspace
  - name: deploy-helm
    taskRef:
      kind: ClusterTask
      name: ods-deploy-helm-v0-2-0
    params:
      - name: release-name
        value: cool-admin-feature
    runAfter:
    - package-image
    workspaces:
    - name: source
      workspace: shared-workspace

In addition to such a preview deployment of an individual repo, there are other repos that I would like to be able to deploy together with an umbrella chart. However the release-name in the umbrella chart is also used in the sub-charts (as expected). However I am not sure how to adjust the image-stream of the subcharts in this scenario.

The reason the image-stream needs to be set in the first place is because the deployment references it as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "chart.fullname" . }}
..
spec:
..
  template:
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "image-registry.openshift-image-registry.svc:5000/{{ .Release.Namespace }}/{{ include "chart.fullname" . }}:{{ .Values.gitCommitSha | default .Chart.AppVersion }}"

And _helpers.tpl brings in the Release.Name for {{ include "chart.fullname" . }}.

One idea that emerged is that deploy-helm could copy the image streams to the destination namespace using a name which is compatible with the above. @michaelsauter what do you think?

michaelsauter commented 2 years ago

One idea that emerged is that deploy-helm could copy the image streams to the destination namespace using a name which is compatible with the above.

I am not sure I am following. Let me expand a little.

Currently, deploy-helm uses existing image digest artifacts, written by package-image, to transport images to the target namespace. It is not consulting the component name or similar because it cannot be known how many images you create for one component.

I am not sure if you really need / want the image stream name to be based on the chart's full name in the first place. You could as well hardcode the name I believe since I do not think it is necessary to have separate image streams per preview. Every Git commit results in one image tag. As such, there is no need for isolation: if you reference the same Git commit, you want the same image tag.

gerardcl commented 2 years ago

hey! happy new year! :) just came around and wanted to propose: shall we do a first meeting and define the use cases we are willing and what is already in place, what is possible, what to nice to have, etc? I think it would be nice to have this as an outcome documented somewhere in the users docs, what do you think? like a "projects structures: folders, architectures, ..."?

henrjk commented 2 years ago

I am not sure if you really need / want the image stream name to be based on the chart's full name in the first place. You could as well hardcode the name I believe since I do not think it is necessary to have separate image streams per preview. Every Git commit results in one image tag. As such, there is no need for isolation: if you reference the same Git commit, you want the same image tag.

Since I posted the comment yesterday I was thinking a bit more about this. Now I am also leaning on not having different streams and simply differentiating by tag. However I still am concerned about the naming clash between an ods-pipeline and the Jenkins pipeline which by default use the same name (component name). So if you are migrating the Jenkins build and the ods-pipeline could potentially interfere with each other. To keep thing separate the default image stream could perhaps be odp-<component-name>.
For example at https://github.com/opendevstack/ods-pipeline/blob/905d3e6952070a6077c72a55dbd5d343f1d0c3f7/cmd/build-push-image/main.go#L108 .

henrjk commented 2 years ago

@gerardcl Happy New Year (again)! By having conversations in GitHub issues and discussions questions can be captured and also giving some direction including for newbie questions. Also some missing features might be identified. At the same time this already is a kind of version 0 of documentation for other users following.

Nonetheless it likely makes sense to have a meeting like you suggested with early adopters.

michaelsauter commented 2 years ago

Since I posted the comment yesterday I was thinking a bit more about this. Now I am also leaning on not having different streams and simply differentiating by tag.

Cool!

However I still am concerned about the naming clash between an ods-pipeline and the Jenkins pipeline which by default use the same name (component name). So if you are migrating the Jenkins build and the ods-pipeline could potentially interfere with each other. To keep thing separate the default image stream could perhaps be odp-component-name.

I'm actually not in favour of this:

I would not want us to change the default just because of the migrating use case.
It is possible to override the image stream so that clashes do not happen.
Following the logic of "simply differentiating by tag", one might not need to separate image streams, even during migration?

@gerardcl Happy New Year to you as well! Yes we need to have a conversation around that, but I am not sure I want to open that can now. For the time being, I would approach it like this:

Everything that is documented is what is supported. Everything that is not documented should not be considered as supported. I strongly recommend reading the whole user guide referenced from https://github.com/opendevstack/ods-pipeline. It does contain a lot of information that will be helpful. I realise though that it is not organised all too well and not searchable. Suggestions for changes highly welcome!
I would encourage everyone to open an issue (or a discussion thread) if you have a certain use case for which you want to use ODS Pipeline but are unsure whether it is possible or how to do it

I believe it will be helpful for us to collect real use cases first before discussing possibilities. ODS Pipeline by design does not do a lot of things that were in discussion at some point in time and I want to be very very cautious before we implement new features.

michaelsauter commented 2 years ago

@henrjk @gerardcl I am going to close this as I believe the original purpose of this issue is fulfilled. Fee free to open follow-up issues to improve docs, add more features, etc. as you see fit for your use case.

opendevstack / ods-pipeline