search

tektoncd / operator

Kubernetes operator to manage installation, updation and uninstallation of tektoncd projects (pipeline, …)
Apache License 2.0
426 stars 186 forks source link

Tekton Operator SCC / UID CreateContainerError #1772

Open ArthurVardevanyan opened 8 months ago

ArthurVardevanyan commented 8 months ago

Expected Behavior

Deployments:

Actual Behavior

Error: container has runAsNonRoot and image will run as root (pod: "openshift-pipelines-operator-ccc7779d6-556lb_openshift-pipelines-operator(7aae737c-ba63-487e-821a-c9b03880c6ef)", container: openshift-pipelines-operator-cluster-operations)

Screenshot from 2023-10-12 21-27-06

Steps to Reproduce the Problem

  1. Install: https://github.com/tektoncd/operator/releases/download/v0.68.0/openshift-release.yaml

Additional Info

Client Version: 4.14.0-rc.5
Server Version: 4.14.0-0.okd-2023-10-12-150331
Kubernetes Version: v1.27.1-3342+98158f96f30cce-dirty

Tekton Pipeline version:

pratap0007 commented 7 months ago

/assign

vdemeester commented 6 months ago

I think we need to update the base image from .ko.yaml to use an image that defines a user (ubi-minimal probably doesn't).

ArthurVardevanyan commented 6 months ago

I don't seem to have an issue with ubi9-minimal /w KO /w OpenShift:

KO: https://github.com/ArthurVardevanyan/openshift-monitoring-cr-controller/blob/production/Makefile#L8 Deployment: https://github.com/ArthurVardevanyan/HomeLab/blob/production/okd/openshift-monitoring/base/openshift-monitoring-cr-controller/deployment.yaml#L51-L59

ArthurVardevanyan commented 3 months ago

The issue appears to be due to the elevated SCCs assigned to the operator pods, it is picking up the AnyUID SCC when it doesn't need it, and OpenShift is no longer injecting the random ID.

Forcing the Deployment to use restriced-v2 takes care of the issue:

Elyytscha commented 3 months ago

same or similar over here when installing the latest tekton operator from operatorhub on okd 4.15

image

vdemeester commented 3 months ago

@ArthurVardevanyan from https://github.com/ArthurVardevanyan/HomeLab/commit/f6a0c235a6d659b7abb7732ec9fc8eb2003c832c, the previous state was pod-security.kubernetes.io/enforce: privileged which is not something the operator sets, anywhere.

Just tried on a brand new cluster, oc apply -f https://github.com/tektoncd/operator/releases/download/v0.68.0/openshift-release.yaml.

NAME                                            READY   STATUS    RESTARTS   AGE
openshift-pipelines-operator-7989dfd96b-fjkl4   2/2     Running   0          9s
tekton-operator-webhook-76c89b7bb7-5rmcb        1/1     Running   0          9s

All just work.

Looking at openshift-operators, it does have it:

  labels:
    kubernetes.io/metadata.name: openshift-operators
    openshift.io/cluster-monitoring: "true"
    openshift.io/scc: ""
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: v1.24

But this is not set by openshift-pipelines, it's probably the default label and is not required for OpenShift Pipelines operator. Also, from your repo, you are deploying it in openshift-pipelines-operator namespace, so you own that namespace, and it's up to you to create it with restricted-v2, it's not something the operator controls.

@Elyytscha which operator from the hub did you install ? If it is https://operatorhub.io/operator/tektoncd-operator it is "tailored" for Kubernetes (e.g. it doesn't remove the specified user if the operand payload such as pipeline, …) and will not work properly on OpenShift (or OKD).

ArthurVardevanyan commented 3 months ago

@vdemeester, Generally I avoid installing anything into the openshift-operators, I have ran into issues in the past, when installing many operators into that single namespace.

Elyytscha commented 2 months ago

@Elyytscha which operator from the hub did you install ? If it is https://operatorhub.io/operator/tektoncd-operator it is "tailored" for Kubernetes (e.g. it doesn't remove the specified user if the operand payload such as pipeline, …) and will not work properly on OpenShift (or OKD).

why its like this? the operator should check if he runs on openshift or not and do the appropriate stuff

the tekton operator is only available via the link you posted, there is no tekton operator in the openshift repos so its the only option to install tekton https://github.com/redhat-openshift-ecosystem/community-operators-prod here is no tekton operator (i think we all agree that OLM is superior and we should avoid helm, kustomize, plain yamls when we can use olm for operator install)

vdemeester commented 1 month ago

why its like this? the operator should check if he runs on openshift or not and do the appropriate stuff

this is not the case today at least. Today there is one build for vanilla k8s and one for openshift (and the latter is supported by red hat). But yeah ideally, we could have one build that adapts, but it would/will require a lot of work/refactoring 😇.

the tekton operator is only available via the link you posted, there is no tekton operator in the openshift repos so its the only option to install tekton

For openshift it is called openshift-pipelines, but it is only available from red hat catalog. I think it is available for OKD but I am not 100% sure of that 😅