debug dev setup failure in CI annotating the ingresses.config

[gabemontero]

See triage in https://github.com/openshift-pipelines/pipeline-service/pull/962#issuecomment-1991758791

@xinredhat @Roming22 FYI

rh-pre-commit.version: 2.2.0 rh-pre-commit.check-secrets: ENABLED

[gabemontero]

So @xinredhat a few things:

1) the permission check in check_cluster_role with --all-namespaces is the incorrect option to check when dealing with cluster scoped objects; hence the check you have there in main branch will not catch the fact that cluster-admin does not seem to be able to annotate ingresses.config

2) for whatever reason the kubectl can-i verb is unware of the annotate verb, even though kubectl annotate is a valid option

3) I was able to work around that by using the oc specific adm policy verbs, and get this list back:

+ oc adm policy who-can annotate ingresses.config
resourceaccessreviewresponse.authorization.openshift.io/<unknown> 

Namespace: default
Verb:      annotate
Resource:  ingresses.config.openshift.io

Users:  backplane-cluster-admin
        system:admin
        system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator
        system:serviceaccount:openshift-authentication-operator:authentication-operator
        system:serviceaccount:openshift-cluster-storage-operator:cluster-storage-operator
        system:serviceaccount:openshift-cluster-version:default
        system:serviceaccount:openshift-config-operator:openshift-config-operator
        system:serviceaccount:openshift-controller-manager-operator:openshift-controller-manager-operator
        system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator
        system:serviceaccount:openshift-kube-controller-manager-operator:kube-controller-manager-operator
        system:serviceaccount:openshift-kube-scheduler-operator:openshift-kube-scheduler-operator
        system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator
        system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa
        system:serviceaccount:openshift-machine-config-operator:machine-config-operator
        system:serviceaccount:openshift-service-ca-operator:service-ca-operator
Groups: cluster-admins
        system:cluster-admins
        system:masters

as system:cluster-admin users is not specifically listed the only inference I can make is that system:cluster-admin is not in any of the cluster-admins group, which on the surface makes no sense to me, but is at lease conceivable

or there is a bug in the ingress-config-validation.managed.openshift.io webhook at whatever version CI is running against

I considered trying to create RBAC to grant permissions to annotate ingresses, but IIRC, there are checks in place to prevent escalation of privileges.

Feels more like more rosa invocations like rosa grant user or rosa create account-roles or something is needed while setting up the cluster-admin role maybe.

Hopefully that is enough for you to engage with whomever you have obtained rosa support from in the past.

[gabemontero]

Entire set of debug:

+ check_cluster_role
+ oc whoami
cluster-admin
+ kubectl auth can-i '*' '*' --all-namespaces
yes
+ oc auth can-i '*' '*' --all-namespaces
yes
++ kubectl auth can-i '*' '*' --all-namespaces
+ '[' yes '!=' yes ']'
+ oc auth can-i annotate ingresses.config
Warning: resource 'ingresses' is not namespace scoped in group 'config.openshift.io'

Warning: verb 'annotate' is not a known verb

yes
+ oc adm policy who-can annotate ingresses.config
resourceaccessreviewresponse.authorization.openshift.io/<unknown> 

Namespace: default
Verb:      annotate
Resource:  ingresses.config.openshift.io

Users:  backplane-cluster-admin
        system:admin
        system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator
        system:serviceaccount:openshift-authentication-operator:authentication-operator
        system:serviceaccount:openshift-cluster-storage-operator:cluster-storage-operator
        system:serviceaccount:openshift-cluster-version:default
        system:serviceaccount:openshift-config-operator:openshift-config-operator
        system:serviceaccount:openshift-controller-manager-operator:openshift-controller-manager-operator
        system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator
        system:serviceaccount:openshift-kube-controller-manager-operator:kube-controller-manager-operator
        system:serviceaccount:openshift-kube-scheduler-operator:openshift-kube-scheduler-operator
        system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator
        system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa
        system:serviceaccount:openshift-machine-config-operator:machine-config-operator
        system:serviceaccount:openshift-service-ca-operator:service-ca-operator
Groups: cluster-admins
        system:cluster-admins
        system:masters

++ oc auth can-i annotate ingresses.config
Warning: resource 'ingresses' is not namespace scoped in group 'config.openshift.io'

Warning: verb 'annotate' is not a known verb

[cluster-setup]
- Enabling HTTP2 for ingress:
+ '[' yes '!=' yes ']'
+ cluster_setup
+ echo '[cluster-setup]'
+ echo '- Enabling HTTP2 for ingress:'
+ oc annotate ingresses.config/cluster ingress.operator.openshift.io/default-enable-http2=true --overwrite=true
+ indent 2
+ offset=2
++ printf %2s
+ sed 's/^/  /'
Error from server (Only privileged service accounts may access): admission webhook "ingress-config-validation.managed.openshift.io" denied the request: Only privileged service accounts may access
command terminated with exit code 1
+ echo 'Failed to execute dev_setup.sh script, retrying ...'
+ sleep 5
Failed to execute dev_setup.sh script, retrying ...