search

gardener / gardener-extension-provider-azure

Gardener extension controller for the Azure cloud provider (https://azure.microsoft.com).
https://gardener.cloud
Apache License 2.0
10 stars 81 forks source link

Azure clusters in existing resource groups #5

Open rfranzke opened 6 years ago

rfranzke commented 6 years ago

Some time back we have disabled deployments into existing Azure resource groups and existing Azure VNets because the Azure cloud provider implementation did not clean up self-created resources properly (tested with version 1.7.6).

Has the Azure cloud provider been improved in that regards so that we can re-enable it again?

Status:

rfranzke commented 6 years ago

cc @dkistner @AndreasBurger

dkistner commented 6 years ago

Currently not. It seems, that there are still issues with orphaned iaas resources after LB service deletion exists. See here: https://github.com/kubernetes/kubernetes/issues/59255 They are already working on a fix for this issue, but I'm not sure if this is the only remaining issue. Further investigation is required.

rfranzke commented 6 years ago

FYI: https://github.com/kubernetes/community/blob/master/keps/sig-azure/0025-20180809-cross-resource-group-nodes.md

rfranzke commented 6 years ago

@dkistner any updates here for Kubernetes 1.{9,10,11,12}?

rfranzke commented 6 years ago

https://github.com/kubernetes/kubernetes/pull/67604 Maybe relevant?

rfranzke commented 5 years ago

@dkistner any update here?

vlerenc commented 5 years ago

Ping... @dkistner.

dkistner commented 5 years ago

I think there is no update as we still see also with latest k8s versions (v1.13, 1.14) left overs in Azure, we would not get rid of them when we delete the cluster. Currently we manage the ResourceGroup and drop it when the Shoot gets deleted. All left overs will be deleted during this process. This would not be the case for existing ResourceGroups because then we would not delete them.

My advise: Wait until we see no left overs anymore and when we have enough trust that the clean up works reliable.

rfranzke commented 5 years ago

@dkistner what I understand from the Kubernetes 1.15 release notes this scenario should become possible, right?

dkistner commented 5 years ago

We are currently testing if the infrastructure left overs are gone with the changes in k8s v1.15. I would recommend to re-enable this only if we are sure that we are really not experience any leaked resources. Let's wait for our test results.

dkistner commented 5 years ago

We will enable Azure Shoot deployments into existing vNets, which are located in a different resource group as the Shoot resources.

Current status:

In general I see with this change no reason why we should re-enable deployments in existing resource groups. We had that only because the Azure Kubernetes provider supported only deployment in existing vNets which are in the same resource group as the cluster resources and we disabled it due to the orphan resource issues (which partly still exists). Meanwhile the Azure Kubernetes provider support also deployments in existing vNets which are in other resource groups (this scenario is now implemented). Therefore I would suggest to remove the inactive validation logic in the Gardener for the existing resource group case and close this issue afterwards. WDYT? @rfranzke, @AndreasBurger, @vlerenc

vpnachev commented 4 years ago

@dkistner can we close this issue ?

dkistner commented 4 years ago

No only the deployment into existing vnets is implemented. The second part deployment into existing resource groups is still in discussion.

vpnachev commented 4 years ago

ok. I have updated the description of this issue and marked the vNETs as implemented.

muenchdo commented 4 years ago

We would be interested in this as we are currently in discussions with a customer about deploying a shoot into their Azure subscription. Being able to create resource groups seems to require quite broad permissions which we would like to avoid.

dkistner commented 4 years ago

As far a I recall correctly the only argument against Shoot deployment into existing resource groups are the left overs which could remain on the infrastructure when the cluster is deleted. Unfortunately the Azure cloud provider have still issues with that. So yes, I think we can discuss to enable that. Btw: PRs for that are warmly welcome :)

muenchdo commented 4 years ago

So you would say it could be enabled again and there would be a disclaimer somewhere that it can result in orphaned resources?

dkistner commented 4 years ago

Yes, something like this I could image. Users have to be aware that some left overs can remain if the cluster is gone.

rfranzke commented 4 years ago

@muenchdo wanna give it a try? If you need help just ping, happy to guide you