[FEATURE] Use Rancher authentication for Harvester

yasker commented 3 years ago

It takes a lot of effort to create an authentication solution, so with the Rancher pre-installed with Harvester (#513), we should try to reuse the Rancher authentication instead of creating Harvester own.

It should be transparent to the user. They shouldn't need to configure Rancher directly to make it work.

guangbochen commented 3 years ago

after enabling the rancher auth provider, the existing imported downstream Harvester will encounter the following issue:

"groups \"github_org://9343010\" is forbidden: User \"system:serviceaccount:cattle-impersonation-system:cattle-impersonation-user-ppg9k\" cannot impersonate resource \"groups\" in API group \"\" at the cluster scope"

And the error logs from the Rancher Server:

2021/07/26 05:16:17 [ERROR] error syncing 'fleet-local': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-readonly, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9ffdae4a-4768-4607-8322-9ee339ca0df2, UID in object meta: , requeuing
2021/07/26 05:16:17 [ERROR] error syncing 'fleet-default': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-readonly, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 2856f008-598f-4af6-9e9f-e565812bab0e, UID in object meta: , requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-local': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-edit": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-edit, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: de4d73d4-e15c-44e0-a0ba-4bcc7fb98c1e, UID in object meta: , requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-local': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-readonly, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c7b469dc-4886-4280-afe6-3d088cb3cdd7, UID in object meta: , requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'cattle-fleet-clusters-system': handler namespace-auth: clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly" already exists, requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-default': handler namespace-auth: clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly" already exists, requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-default': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-readonly, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 140dd5d3-42c6-47ac-9611-3ceee5fb28d1, UID in object meta: , requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-default': handler namespace-auth: Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly": StorageError: invalid object, Code: 4, Key: /registry/clusterroles/p-j2467-namespaces-readonly, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 140dd5d3-42c6-47ac-9611-3ceee5fb28d1, UID in object meta: , requeuing
2021/07/26 05:16:18 [ERROR] error syncing 'fleet-local': handler namespace-auth: clusterroles.rbac.authorization.k8s.io "p-j2467-namespaces-readonly" already exists, requeuing

gitlawr commented 3 years ago

@guangbochen It is an upstream issue: https://github.com/rancher/rancher/issues/33798

guangbochen commented 3 years ago

To Test:

Should verify using the latest rancher v2.6.0 release and the latest harvester ISO image.

Verification scope:

Either install Rancher 2.6.0 via the docker run or using your existing Rancher server
From the Rancher UI, click the Virtualization Management tab and select importing existing cluster to import the downstream harvester cluster to the Rancher server
the cluster should be imported and can be accessed properly
enable the auth provider(e.g the Github auth provider)
login with auth provider and verify the account permission is correct, the cluster should be healthy and VM-related features should be working properly.
- verify the other authorized users was able to log in and have the correct permission in regards to https://github.com/harvester/harvester/issues/582

shangma commented 2 years ago

Test Info

Verified version:
- Rancher: v2.6.1-rc10
- Harvester: v0.3.0-rc1
Environment: rancher and harvester are running in its own vm respectively

Test case

start rancher using docker in a vm and start harvester in another
import harvester into rancher from "Virtualization Management" page
On rancher page, go to "User & Authentication" -> "Auth Provider" and choose Github
Follow the instructions on the page and eventually copy over client id and client secrets from github
Log out rancher and see a new option "Log in with Github". Click on it to log in.
Go to harvester cluster from "Virtualization Management" page, and double check the cluster is healthy and I am able to create a vm

johnwc commented 5 months ago

This is a bit confusing. When you state Rancher, do you refer to the rancher that is embedded into Harvester, or a separate install of Rancher that is using harvester as it's Virtualization Management?

We are needing to have harvester login utilize Azure AD, is this just as simple as going into the Harvester embedded rancher UI and configuring Azure AD authentication provider?

tserong commented 2 months ago

This is a bit confusing. When you state Rancher, do you refer to the rancher that is embedded into Harvester, or a separate install of Rancher that is using harvester as it's Virtualization Management?

AFAIK it means a separate install of Rancher that has imported the Harvester cluster. There's some documentation about this at https://docs.harvesterhci.io/v1.3/rancher/virtualization-management#multi-tenancy. I haven't tried this myself, but AIUI, you'd hook that Rancher up to use Azure as its auth provider, then assign users/groups to the appropriate cluster and/or project roles.

johnwc commented 2 months ago

Why would we not just enable the auth provider within the existing embedded rancher for harvester? I don't understand the need to import harvester's rancher as a downstream rancher of and existing rancher install.

tserong commented 2 months ago

Because harvester's use of its embedded rancher is a restricted use case; we don't want to expose all of its functionality. There's actually work in progress to simplify Harvester's dependencies (see https://github.com/harvester/harvester/issues/5278) which may involve removing or replacing parts of the embedded rancher.

johnwc commented 2 months ago

But we should not have install Rancher just to be able to use auth providers for harvester. We should be able to enable the auth provider within harvester, which intern enables it for the embedded Rancher.

ibrokethecloud commented 2 months ago

The in built rancher is only mean for bootstrapping harvester and subsequent upgrade cycles. It does have a few features disabled. We recommend against using it for auth and management of more downstream clusters. Among other things this rancher is tied to the harvester packaging and that makes out of band upgrades of rancher difficult as you could end up in unexpected scenarios. If you are really keen on running rancher without VM's on harvester, we do recommend using the rancher-vcluster addon.

johnwc commented 2 months ago

I think there is some confusion. I am not referring to using the embedded rancher for anything downstream. I am asking about using an auth provider to login to the harvester UI, instead of having to create all admins as harvester users.

If you are really keen on running rancher without VM's on harvester, we do recommend using the rancher-vcluster addon.

I have no idea what you are trying to say here. How did we get to talking about running Rancher not in harvester? We have a few harvester clusters that are running nothing but ordinary VMs, no Rancher VMs installed in the cluster at all. Just simple VM cluster management.

tserong commented 2 months ago

What we're trying to say is that the way the Harvester project ultimately decided to support external authentication providers is via a separate Rancher instance that has imported the Harvester cluster, and that you cannot use the Rancher UI embedded inside Harvester to enable external authentication providers directly.

Here's a screenshot of the embedded Rancher UI:

Note that it's only showing cluster resources. There's no bar down the left where you'd ordinarily find the link to the Users & Authentication screen where you'd go to set up an external authentication provider in Rancher proper.

If you don't wish to run a Rancher instance elsewhere in order to configure external auth for a given Harvester cluster, it's possible to run a separate Rancher instance directly on the Harvester cluster itself using the rancher-vcluster addon as mentioned above. You'd import the Harvester cluster(s) into that rancher instance, and have that Rancher instance configured to talk to an external authentication provider. Your users would then login to that Rancher instance and access Harvester via the Virtualization Management interface.

johnwc commented 2 months ago

@tserong correct, I am pointing out that it is a huge oversight to force an install of Rancher in order to get SSO within harvester. Especially when the underlining technology that harvester is running on top of, already supports such a functionality.

tserong commented 2 months ago

It's not an oversight, it was a deliberate choice.

Using a separate Rancher instance has other benefits, notably multi-cluster management. You mentioned earlier that you have multiple Harvester clusters. If you also ran one Rancher instance, you'd be able to hook that Rancher instance up to your authentication provider, then use it to provide SSO for all your Harvester clusters, via a single pane of glass.

If you'd rather not do that, and would prefer to configure SSO for each Harveser cluster separately, you can currently do so via rancher-vcluster.

Further discussion probably really belongs on https://github.com/harvester/harvester/issues/4024

harvester / harvester

[FEATURE] Use Rancher authentication for Harvester #552

Test Info

Test case