[ROSA] MAS deployment via Tekton Pipeline - service account lacks permissions

[Daniel-M-Jacobson]

In both ROSA and ROSA HCP, there are webhooks that disallow creating objects in certain namespaces. "Prevented from accessing Red Hat managed namespaces."

UDS_403Forbidden.txt

After some discussion with a red hat architect, I was pointed to this code https://github.com/openshift/managed-cluster-validating-webhooks/blob/master/pkg/webhooks/namespace/namespace.go#L220

When deploying MAS on pipeline, the service account does not have sufficient permissions to install MAS because of namespace choices.

I was able to successfully complete the same task by running installation from the ansible playbook as cluster-admin.

[durera]

The CLI sets up the pipelines to run under a serviceaccount with admin permissions ... in the logs provided it's complaining that the catalogsources can't be created in openshift-marketplace, did the red hat architect provide any information on how to allow a service account to work in these "managed namespaces", it sounds like you could do so just fine as a cluster-admin, but that's the same permission the service account we're running the pipelines under should have.

I don't specifically have access to ROSA or ROSA HCP to test I'm afraid, but I'd expect the pipelines service account to have the permissions you had when you ran as cluster-admin due to this ClusterRoleBinding:

# 1. Configure RBAC for the pipeline tasks
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ibm-mas-pipeline-{{mas_instance_id}}
subjects:
  - kind: ServiceAccount
    name: pipeline
    namespace: mas-{{mas_instance_id}}-pipelines
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin

[Daniel-M-Jacobson]

There wasn't too much more info provided besides that link to their codebase, and I've destroyed and re-created the cluster that originally had this issue. It seems like it will be hard to reproduce and I don't want to discount that possibility that I butchered something on that deployment. With a workaround available, it might make sense to tag it "cannot reproduce" and see if its eventually reported by anyone else. I won't have time to stand up another environment to test this. If you want to look further...

it looks like there's a regex they use - not sure if you would be allowed to name your service account to match that https://github.com/openshift/managed-cluster-validating-webhooks/blob/master/pkg/webhooks/utils/utils.go#L25

    PrivilegedServiceAccountGroups string = `^system:serviceaccounts:(kube.*|openshift.*|default|redhat.*|osde2e-(h-)?[a-z0-9]{5})`

Here seems to be where the validation method starts, if seems if your sa name matches regex it seems like it might work but I suspect we're not supposed to use that.

https://github.com/openshift/managed-cluster-validating-webhooks/blob/master/pkg/webhooks/namespace/namespace.go#L187

// service accounts making requests will include their name in the group
    for _, group := range request.UserInfo.Groups {
        if privilegedServiceAccountsRe.Match([]byte(group)) {
            ret = admissionctl.Allowed("Privileged service accounts may access")
            ret.UID = request.AdmissionRequest.UID
            return ret
        }
    }

the other way is to try to pass this check that is done right before throwing the error. I don't know of the intricacies of service accounts and request user info and if they are passed properly, but I'd expect your config to work and match the group portion of the check.

clusterAdminGroup            string = "cluster-admins"
clusterAdminUsers           = []string{"kube:admin", "system:admin", "backplane-cluster-admin"}

func amIAdmin(request admissionctl.Request) bool {
    if slices.Contains(clusterAdminUsers, request.UserInfo.Username) || slices.Contains(request.UserInfo.Groups, clusterAdminGroup) {
        return true
    }

    for _, group := range sreAdminGroups {
        if slices.Contains(request.UserInfo.Groups, group) {
            return true
        }
    }

    return false
}

[durera]

Ahhhh, I see the issue ... i'm not overly familiar with groups in OCP as never really had a reason to use them ... but roles and groups are two different things.

Currently we are giving the service account the cluster role cluster-admin, and usually that's enough. That's not the same as adding the service account to a group named cluster-admins.

I know how to add users to a group, but I don't know if it works the same way with serviceaccounts .. we'll have to investigate this further, but it's definitely something we should either handle automatically or at minimum provide some documentation on extra steps needed on ROSA.

[Delgon]

Or the installation shouldn't use the namespace redhat-marketplace which is one of the names reserved by redhat on ROSA. As we can see in the log :

Error from server (Prevented from accessing Red Hat managed namespaces. Customer workloads should be placed in customer namespaces, and should not match an entry in this list of regular expressions: 
[^kube.*
^default$
^redhat.*
^dedicated-admin$
^openshift-addon-operator$
^openshift-
...

[Delgon]

Solved with the release 9.4.0. We can now choose the namespace. Tested with succes on my ROSA.