[BUG] custom cluster roles not available in downstream harvester cluster

[ibrokethecloud]

Rancher Server Setup

• Rancher version: v2.6.11
• Installation option (Docker install/Helm Chart): helm
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): k3s
• Proxy/Cert Details: letsencrypt

Information about the Cluster

• Kubernetes version: v1.24.7
• Cluster Type (Local/Downstream): Harvester v1.1.1+
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

    A custom role created in rancher is not propagated to managed harvester clusters.

    For example in harvester we have additional cluster scoped crds: pcidevices.devices.harvesterhci.io

    A custom role created for pcidevices, doesn't get propagated to downstream harvester cluster.

    If same role is assigned to a cluster member on a non harvester cluster, like a k3s cluster, the role is propagated correctly.

    To Reproduce

    Create a custom cluster scoped role on rancher. In this case the role pcidevices provides the following access:

    

    This role is assigned to a user on two clusters:

    k3s cluster assignment:

    harvester cluster assignment:

    Result When logged in as the test user, I can query pcidevices on the k3s cluster just fine:

    

    However on the harvester cluster this fails:

    If i query both downstream clusters using my admin credentials I can see the role rt-x5xpg has been created correctly on the downstream k3s cluster but not on the harvester cluster.

    k3s cluster role query:

    harvester cluster role query:

    Expected Result

    Expected role to be propagated to downstream harvester cluster as well.

    Screenshots

    Additional context

[samjustus]

/backport 2023-Q2-v2.7x test

[samjustus]

SURE-6009

[JonCrowther]

I have attempted multiple ways to reproduce this bug with no success.

@ibrokethecloud , please let me know if I am not following the steps correctly, but here is what I do:

1. With my test user jono-test that has the role Cluster Member for my harvester cluster I attempt to get pcidevices and see I don't have permission:

2. I create a cluster role that inherits from Cluster Member that has all the verbs for * Resource in API Group devices.harvesterhci.io

3. I then add my test user (which has no other permissions) to my harvester cluster with the Member Role of pcidevices

Note: My user is a User-Base

4. I log in with my test user, then in my harvester cluster I open the kubectl shell and do k get pcidevices and am successful

Some other things I have attempted that were all successful and behaved as expected

• Edit an existing custom cluster role to add new permissions
• Create a custom role cluster that does not inherit from Cluster Member
• Adding specific resources, not just *
• Tried different API Groups
• Grant multiple different resources at the same time

[JonCrowther]

Some information that could help:

• What user created the cluster roles? Were they the admin?
• Does the harvester cluster have errors in the logs?
• What kind of Global Permissions does the test user have? (Cluster Member, Base User, etc)
• What's the status of the harvester cluster on the UI? (is the agent connected?)
• The full yaml of the cluster role (kubectl get roleTemplate <role name> -o yaml)
• Logs from rancher. If it can't create the downstream role there should be logs indicating an issue repeatedly.

[ibrokethecloud]

what is the version of rancher being used? I can share the details of my environment.

[JonCrowther]

The rancher version is v2.7.4 and harvester is v1.1.2

[ibrokethecloud]

The reason for no error was the lack of actual pcidevice objects being created in the harvester cluster.

I created some pcidevice objects and now the error is re-producible in the environment.

[JonCrowther]

I was able to create the error temporarily. However, as soon as I added the following role to a user: I was no longer able to recreate the error. Every role I attempted to add afterwards would successfully get propagated to the harvester cluster (including the existing role that had previously given an error)

I'm going through the logs to try and identify what happened, but this may function as a temporary workaround? Admittedly I can only get it to happen once since the error is gone now, so I'm not sure how consistent this may be.

[samjustus]

moving teams to harvester cc @rebeccazzzz

[Jono-SUSE-Rancher]

@rebeccazzzz - I haven't seen or heard any movement on this for v2.9.0 or v2.9.1. I am going to move it to v2.9-Next2, please let me know if you have a corresponding Harvester issue and maybe we can close this one?