custom tekton pipeline parameters get overwritten by lighthouse

[gerrnot]

intro

Hi, we are using lighthouse (standalone, without the rest of jenkins-x) in combination with tekton. So far the integration to our Git Servers (Bitbucket Server + GitHub) is working good enough - thanks for that! (ChatOps not working on Bitbucket Server, but we can live with that - another story).

On the other hand, there seems to be a strange bug when using a custom tekton pipeline (that defines its own parameters)

Notice: steps to reproduce below use lighthouse config, full config given below...

in_repo_config:
  enabled:
    'PEP': true # pep is the project where my repo resides

short steps to reproduce

1. Lets say you define kind: Pipeline with custom param sshGitRepositoryUrl. (Notice lighthouse seems to add additional parmetes like REPO_URL too - which is fine!)
2. Now you create a kind: PipelineRun yaml file (that is referenced in triggers.yaml) in the source code and specify a concrete custom value as param, e.g.:

   params:
    - name: sshGitRepositoryUrl
      value: ssh://git@mybitbucketserverhost.com:7999/pep/tkn-demo-go-bb.git

3. git push in the repo from step 2.

long steps to reproduce

(this is just supplementary material to short steps to reproduce)

additional information is provided with detailed sources/configs:

1. source code repo of the application that contains the .lighthouse folder\ tkn-demo-go-bb.tar.gz
2. tekton pipeline definition can be found here Hint: in ../test/local.sh there is a local.sh that can be used with minimal adaptation to set up this experimental pipeline.
3. lighthouse config\ config.yaml:

   
   # docu link: https://github.com/jenkins-x/lighthouse/blob/main/docs/config/lighthouse/github-com-jenkins-x-lighthouse-pkg-config-lighthouse.md

pod_namespace: lighthouse tkn-demo-go-bb.tar.gz

prowjob_namespace: lighthouse

in_repo_config: enabled: 'PEP': true # must have been uppercase here

tide: queries:

• labels:
  • approved repos:
  • PEP/tkn-demo-go-bb

    plugins.yaml:

    approve:

  • lgtm_acts_as_approve: false repos:
• pep/tkn-demo-go-bb require_self_approval: true config_updater: gzip: false maps: config.yaml: name: config plugins.yaml: name: plugins triggers:
  • repos:
• PEP ignore_ok_to_test: false elide_skipped_contexts: false plugins:

  PEP/tkn-demo-go-bb: # whitelisted entire pep space below

  - trigger

  PEP:

• trigger

  # actual result
  the resulting PipelineRun manifest (created by lighthouse via a git push event) looks like this (shortened):

  apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"tekton.dev/v1beta1","kind":"Pipeline","metadata":{"annotations":{},"labels":{"app.kubernetes.io/version":"0.1.0"},"name":"cd-pipeline","namespace":"lighthouse"},"spec":{"description":"from build to production!","params":[{"default":"","name":"BUILD_ID"},{"default":"","name":"JOB_NAME"},{"default":"","name":"JOB_SPEC"},{"default":"","name":"JOB_TYPE"},{"default":"","name":"PULL_BASE_REF"},{"default":"","name":"PULL_BASE_SHA"},{"default":"","name":"PULL_REFS"},{"default":"","name":"REPO_NAME"},{"default":"","name":"REPO_OWNER"},{"default":"","name":"REPO_URL"},{"default":"","description":"git repo url as used by git clone command","name":"sshGitRepositoryUrl"}],"tasks":[{"name":"git-clone","params":[{"name":"url","value":"$(params.sshGitRepositoryUrl)"},{"name":"revision","value":"$(params.PULL_BASE_REF)"},{"name":"depth","value":"0"}],"taskRef":{"name":"git-clone"},"workspaces":[{"name":"output","workspace":"source-ws"},{"name":"ssh-directory","workspace":"ssh-creds"}]},{"name":"git-checkout","params":[{"name":"revision","value":"$(params.PULL_BASE_REF)"}],"runAfter":["git-clone"],"taskRef":{"name":"git-checkout"},"workspaces":[{"name":"gitrepo","workspace":"source-ws"},{"name":"ssh-directory","workspace":"ssh-creds"}]},{"name":"version-calc","params":[{"name":"branch","value":"$(params.PULL_BASE_REF)"}],"runAfter":["git-checkout"],"taskRef":{"name":"git-version"},"workspaces":[{"name":"source","workspace":"source-ws"}]},{"name":"version-print","params":[{"name":"string","value":"$(tasks.version-calc.results.gitVersion)"}],"runAfter":["version-calc"],"taskRef":{"name":"echo"}},{"name":"add-overlay-workspace-to-source","params":[{"name":"script_source","value":"#!/bin/env bash\nset -ex\ncp -rT /workspace/workspace-a /workspace/workspace-c\nif [ \"$(workspaces.source-ws-overlay.bound)\" == \"true\" ];\nthen\n cp -rT /workspace/workspace-b /workspace/workspace-c\nfi\n"}],"runAfter":["git-checkout"],"taskRef":{"name":"bash"},"workspaces":[{"name":"workspace-a","workspace":"source-ws"},{"name":"workspace-b","workspace":"source-ws-overlay"},{"name":"workspace-c","workspace":"source-ws-overlaid"}]},{"name":"docker-build","params":[{"name":"IMAGE","value":"phx-temp-registry.myartifactoryhost.com/pep/$(params.REPO_NAME):$(tasks.version-calc.results.gitVersion)"},{"name":"EXTRA_ARGS","value":["--build-arg=APP_VERSION=$(tasks.version-calc.results.gitVersion)"]}],"runAfter":["add-overlay-workspace-to-source","version-calc"],"taskRef":{"name":"kaniko"},"workspaces":[{"name":"source","workspace":"source-ws-overlaid"},{"name":"dockerconfig","workspace":"docker-config"}]},{"name":"helm-build","params":[{"name":"script_source","value":"echo \"to be implemented yet\"\n"}],"runAfter":["add-overlay-workspace-to-source","version-calc"],"taskRef":{"kind":"Task","name":"bash"}},{"name":"helm-deploy","params":[{"name":"script_source","value":"echo \"to be implemented yet\"\n"}],"runAfter":["helm-build","docker-build"],"taskRef":{"kind":"Task","name":"bash"}},{"name":"git-tag","params":[{"name":"GIT_SCRIPT","value":"git tag '$(tasks.version-calc.results.gitVersion)'\ngit push origin '$(tasks.version-calc.results.gitVersion)'\n"}],"runAfter":["helm-deploy"],"taskRef":{"name":"git-cli"},"when":[{"input":"$(params.PULL_BASE_REF)","operator":"in","values":["master","main","production"]}],"workspaces":[{"name":"source","workspace":"source-ws"},{"name":"ssh-directory","workspace":"ssh-creds"}]}],"workspaces":[{"description":"Location where source is cloned/checked out to","name":"source-ws"},{"description":"Optional Workspace containing files/folders required for building that are typically not part of the source code. Examples: maven settings.xml, jfrog-cli.conf). The contents of source-ws-overlay will be copied to source-ws-overlaid, if provided.\n","name":"source-ws-overlay","optional":true},{"description":"source-ws-overlaid = source-ws + source-ws-overlay (this is done by task add-overlay-workspace-to-source)\n","name":"source-ws-overlaid"},{"description":"git ssh credentials - see https://hub.tekton.dev/tekton/task/git-clone","name":"ssh-creds"},{"description":"docker config as required by https://hub.tekton.dev/tekton/task/kaniko","name":"docker-config"}]}} lighthouse.jenkins-x.io/cloneURI: http://mybitbucketserverhost.com/scm/pep/tkn-demo-go-bb.git lighthouse.jenkins-x.io/job: release creationTimestamp: "2022-05-19T10:29:08Z" generateName: ep-tkn-demo-go-bb-master-release- generation: 1 labels: app.kubernetes.io/version: 0.1.0 created-by-lighthouse: "true" event-GUID: e39f9a16-9a19-45b3-bc5e-c545961f3657 lighthouse.jenkins-x.io/baseSHA: 31105ca37caa431a19173b32593a91328cb9714f lighthouse.jenkins-x.io/branch: master lighthouse.jenkins-x.io/buildNum: "1652956030682" lighthouse.jenkins-x.io/context: release lighthouse.jenkins-x.io/id: ep-tkn-demo-go-bb-master-release-nhnhg lighthouse.jenkins-x.io/job: release lighthouse.jenkins-x.io/lastCommitSHA: 31105ca37caa431a19173b32593a91328cb9714f lighthouse.jenkins-x.io/refs.org: pep lighthouse.jenkins-x.io/refs.repo: tkn-demo-go-bb lighthouse.jenkins-x.io/type: postsubmit tekton.dev/pipeline: cd-pipeline name: ep-tkn-demo-go-bb-master-release-bk27c namespace: lighthouse ownerReferences:

• apiVersion: lighthouse.jenkins.io/v1alpha1 controller: true kind: LighthouseJob name: ep-tkn-demo-go-bb-master-release-nhnhg uid: 9c6e89ed-ce58-4c00-a227-78b0b94597df resourceVersion: "712253089" uid: b8fd0d4d-57e3-4fb5-b913-541753171188 spec: params:
• name: gitMainBranch value: master
• name: sshGitRepositoryUrl value: ssh://git@mybitbucketserverhost.com:7999/pep/tkn-demo-go-bb.git
• name: BUILD_ID value: ""
• name: JOB_NAME value: ""
• name: JOB_SPEC value: ""
• name: JOB_TYPE value: ""
• name: PULL_BASE_REF value: ""
• name: PULL_BASE_SHA value: ""
• name: PULL_REFS value: ""
• name: REPO_NAME value: ""
• name: REPO_OWNER value: ""
• name: REPO_URL value: ""
• name: BUILD_ID value: "1652956030682"
• name: JOB_NAME value: release
• name: JOB_SPEC value: type:postsubmit
• name: JOB_TYPE value: postsubmit
• name: PULL_BASE_REF value: master
• name: PULL_BASE_SHA value: 31105ca37caa431a19173b32593a91328cb9714f
• name: PULL_REFS value: master:31105ca37caa431a19173b32593a91328cb9714f
• name: REPO_NAME value: tkn-demo-go-bb
• name: REPO_OWNER value: PEP
• name: REPO_URL value: http://mybitbucketserverhost.com/scm/pep/tkn-demo-go-bb.git
• name: sshGitRepositoryUrl value: http://mybitbucketserverhost.com/scm/pep/tkn-demo-go-bb.git pipelineRef: name: cd-pipeline ...

  
  Notice that the param `sshGitRepositoryUrl` occurs twice, first with the correct value
  (`ssh://git@mybitbucketserverhost.com:7999/pep/tkn-demo-go-bb.git`, then with the wrong value
  `http://mybitbucketserverhost.com/scm/pep/tkn-demo-go-bb.git`). Unfortunately, the latter (=wrong) parameter wins.

expected result

The resulting PipelineRun should contain the custom parameter only once and with the expected value:

  - name: sshGitRepositoryUrl
    value: ssh://git@mybitbucketserverhost.com:7999/pep/tkn-demo-go-bb.git

Lighthouse should leave custom parameters (and their values) as they are: There should not be an interference (lighthouse parameter values copied into custom parameter values).

[gerrnot]

analysis After debugging, I figured out the problem:

There exists a method (determineGitCloneOrMergeTaskParams) that performs some special magic (among similar magic for similar things):

• Do you use the git-clone task?
• Do you pass a pipeline param to the task git-clone
• If so, I believe that this parameter is your git url parameter and I (lighthouse) will fill it with the value that I calculated (=the git http url that is also used for pipeline parameter REPO_URL) and override your custom value that you specified in the PipelineRun.

workaround: I swapped the git-clone task to a custom implementation and named it git-checkout (idempotent checkout(+clone) so to speak), then lighthouse does not touch my parameters.

What could be done better? I do not understand why the method determineGitCloneOrMergeTaskParams reads the url parameter and overwrites the value of it, while it actually pretends to known the url better anyway (existing parameter REPO_URL). So why touch further parameters?. However, as the codebase is quite complex and it is hard to know about all downstream effects, I would rather not touch it for now. But closing also seems weird, as I have the feeling the other people could also fall into this pit. Maybe some jenkins-x developer can give a comment and decide whether it is "okay enough" to leave it as it is and perhaps decide to close....