How to move system deployed in tenant A to tenant B without recreating?

[m1n9o]

As you can see, I have two tenants, blue tenant has a GitRepository which points to repo "blue". And inside the "blue" repo, I have a deployment which to deploy my application. In our company, we may have reorg sometimes, which requires us to change the custodianship of the system. Which means, we may move a system from Tenant A (team A) to Tenant B (team B).

So how to move the system without re-creating with flux multi-tenant solution?

> $ tree . -L 3                                                                                                                                                                                                      
.
├── README.md
├── blue-secret.yaml
├── clusters
│   └── minikube
│       └── flux-system
├── flux-system-secret.yaml
└── tenants
    ├── blue-team
    │   ├── blue-secret.yaml
    │   ├── blue-team-git.yaml
    │   ├── blue-team-tenant.yaml
    │   ├── flux-kustomize.yaml
    │   └── kustomization.yaml
    └── green-team
        ├── flux-kustomize.yaml
        ├── green-secret.yaml
        ├── green-team-git.yaml
        ├── green-team-tenant.yaml
        └── kustomization.yaml

[stefanprodan]

Kubernetes does not allow for some Deployment to be moved from one namespace to another, you can only delete/recreate. The tenant boundary is the namespace, so i don't see how would you move some app from one tenant to another without recreating it.

[m1n9o]

Yeah, understood, pods can not be moved from ns A to ns B. Let me ask in this way, what it will happen if you move blue-team-tenant.yaml (RoleBinding, ServiceAccount, Namespace) , blue-team-git.yaml (GitRepository) and flux-kustomize.yaml (Kustomization) to tenant green-team? Will every resource be recreated?

[stefanprodan]

@m1n9o see here how to move things around: https://fluxcd.io/flux/faq/#how-can-i-safely-move-resources-from-one-dir-to-another

[m1n9o]

I have another scenario, our tenant is tribe level, which means there would be more than one namespace under a tenant. So user's belongs to this tenant should be able to create namespace, edit and delete namespace. But how to prevent user from tenant A deleting namespaces that belongs to other tenant?

[stefanprodan]

You would need RBAC based on label selectors which is not a thing, I think Kubernetes is not a good fit for your tenancy model. So either you move the namespace responsibility to the admin team, or move to hard multi-tenancy where each tenant has its own clusters, or move to some other system that's not Kubernetes.

[m1n9o]

Thanks for your timely response, I see. To my understanding, we may need to develop our own admission controller to do that based on label. Is that what you mean?

You would need RBAC based on label selectors which is not a thing

[stefanprodan]

You could use the builtin admission from Kubernetes 1.30, you can block namespace operations such as edit/delete with CEL: https://fluxcd.control-plane.io/guides/flux-policies/

[m1n9o]

Do you have an example to block namespace operation? I found it's hard to write one by myself.

[stefanprodan]

I don't, this will be a tricky one as RBAC is not meat for that, I really don't want to encourage people on rolling their own RBAC extension which breaks the namespace boundary. For each tenant you could create a dedicated policy scoped to its SA in some namespace that blocks UPDATE and DELETE for other namespaces based on some label, but be careful as this has the potential to lock out the cluster admins.