Problems I have when installing JX3 on a kubeadm cluster

[aure-olli]

This is a list of the troubles I had when trying to install jx3 on a cluster created with kubeadm.

• lack of configuration on ingress-nginx: this is not a big trouble, can be fixed by directly editing versionStream/charts/ingress-nginx/ingress-nginx/values.yaml.gotmpl, but cannot have advanced configuration on ingress-nginx in requirements.yaml, like the Daemonset/hostPort option with specific nodeSelector/tolerations , which makes much sense on kubeadm cluster behind a laod-balancer. This was also the case in jx2 anyways.
• jx-pipelines-visualizer doesn't use the given secretName: in versionStream/charts/jx3/jx-pipelines-visualizer/values.yaml.gotmpl, secretName is not taken in account, it will always generate a secret name, whereas other charts work fine (https://github.com/jx3-gitops-repositories/jx3-kubernetes/blob/main/versionStream/charts/jx3/jx-pipelines-visualizer/values.yaml.gotmpl#L16)
• Cannot customize the docker configuration: in jx-global-values.yaml there is a imagePullSecrets listing secrets use to configure docker. Editing that doesn't seem to have any effect. (https://github.com/jx3-gitops-repositories/jx3-kubernetes/blob/main/jx-global-values.yaml#L4)
• the docker secret used by kaniko is always empty: jxboot-helmfile-resources will always create the github and docker secrets, no way to use your own. for github, it is filled with the user/password provided on installation, but for docker, the secret is always empty. I cannot prevent the secret from being created (and put mine instead), I cannot force to use another secret, only way would be to provide the configJson in the values. I don't understand exactly how secret-schema.yaml works, I tried to use it to copy my existing secrets, but it doesn't seem to have any effect on my installation. (https://github.com/jenkins-x-charts/jxboot-helmfile-resources/blob/master/charts/jxboot-helmfile-resources/templates/tekton-container-registry-auth-secret.yaml)

I spent a week trying to have a clean configuration, that would not be broken every time the dev environment is redeployed, that last problem made me give up. I preferred installing a custom jx2 using helm3 for templates.

[jstrachan]

• for customizing nginx you can configure the chart in the usual helmfile way to do pretty much anything you need. We only use jx-requrements.yml for common settings (like domain name that is used across many charts)
• sounds like an improvement we should make to allow the ingress secret to be reused - see https://github.com/jenkins-x/jx3-versions/pull/2460
• I don't understand what you mean by having no effect? no effect on what? using image pull secrets when deploying apps? Depends on the chart of the app you deploy as to whether it looks for those values - since the image pull secrets are inside helm chart's Deployment resources usually
• to setup custom registry secrets for kaniko see: https://jenkins-x.io/v3/develop/faq/config/registries/#how-do-i-use-a-custom-container-registry