search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.97k stars 306 forks source link

AKS Run Command - Simplified kubectl access to private cluster #2059

Closed palma21 closed 3 years ago

palma21 commented 3 years ago

https://docs.microsoft.com/en-us/azure/aks/private-clusters#aks-run-command-preview

ghost commented 3 years ago

@Azure/aks-pm issue needs labels

lgmorand commented 3 years ago

That looks awesome. Any idea of the implementation ? How would it work, like what would be the user experience of that ?

ghost commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

ghost commented 3 years ago

This issue will now be closed because it hasn't had any activity for 15 days after stale. palma21 feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.

palma21 commented 3 years ago

https://docs.microsoft.com/en-us/azure/aks/private-clusters#aks-run-command-preview

ohorvath commented 3 years ago

That's a pretty useful feature, thank you. Are you going to support service principal based AKS clusters in the GA version?

jabbera commented 3 years ago

We need SP based support as well,

palma21 commented 3 years ago

Should work on any kind of clusters SPN or managed identity (but I do recommend you migrate from SP to managed identity, should be quick and painless otherwise let us know!)

ohorvath commented 3 years ago

@palma21 I tried again and it's failing with both SP and with User Assigned identities. I have Azure AD integration and RBAC. Also the AKS cluster creation is broken with user assigned identities, I've opened a support ticket for that part.

miwithro commented 3 years ago

@haitch can you look into this?

haitch commented 3 years ago

@ohorvath this is a internal error we are hitting, it is already fixed, our deployment pipeline will take another week to reach to westeurope.

ohorvath commented 3 years ago

@haitch Thanks for the update. Which error are you referring to? We have multiple issues currently. :) And I have my resources in westus2, centralus, so if the fix is there, I can try again.

ohorvath commented 3 years ago

I've checked this again and I can repro the issue. AKS clusters with BYOD user assigned identities work fine with the run command. However AAD integration breaks this feature.

Failed to connect to MSI. Please make sure MSI is configured correctly. Get Token request returned: <Response [400]>

I've changed nothing but the AAD integration.

edit:

It seems I had to run AZ login manually to get the token. This should be mentioned in the docs. Now I have another error message but at least this one has some information.

Operation failed with status: 'Bad Request'. Details: Run command is not supported when both Private Cluster and AAD are enabled

That's so sad as all of our production clusters has AAD integration, I can't imagine any enterprise customers use AKS without AAD especially if they need private AKS for security reasons. So this feature is not really helpful to manage private clusters.

haitch commented 3 years ago

@ohorvath I am pretty sure you are using in azure cloud shell. This is a known issue from cloud shell, I can find you the link to this issue.

haitch commented 3 years ago

here is the issue tracking from cloudshell side:

https://github.com/Azure/azure-cli/issues/11749 https://github.com/Azure/azure-cli/issues/17695

shreyasHpandya commented 3 years ago

Is it possible to call k8 API via this feature?

haitch commented 3 years ago

@shreyasHpandya you can run kubectl and helm in this cli, kubectl and helm invokes k8s API behind the sense.

so it is possible.

but my question is why? this feature is designed for cluster admin to run simple, one off command without worrying isolated network (given you are authorized in ARM and have permission).

If you are going to have some app and work with k8s API, then why bother going through ARM, you can directly deploy this app to your k8s cluster.

shreyasHpandya commented 3 years ago

Let's say I am developing a third-party CSPM tool that wants to scan their cluster for misconfigurations etc. my client has already given me access to their azure API, I want to use the same SP to extend functionality to AKS

haitch commented 3 years ago

that's not a designed scenario.

you should publish your app somewhere, then ask your client to install your app.

shreyasHpandya commented 3 years ago

Yes we do that currently, but this way we can reduce the pain of onboarding for client, If possible consider this kind of use case, people will eventually use it for this I guess

haitch commented 3 years ago

for a normal cluster, if your client give you enough access to their AKS resource, then you should be able to call kubernetes API.

for a private cluster:

mikebranstein commented 3 years ago

I'm in agreement with @haitch. We have private clusters for a reason - we want to restrict access to internal networks only. I'm not opposed to this feature, but I wouldn't want the feature to be enabled by default.

for a normal cluster, if your client give you enough access to their AKS resource, then you should be able to call kubernetes API.

for a private cluster:

  • If you client give you access to both AKS resource and vNet resource, then you should also be able to call Kubernetes API from that vNet.
  • If your client give you only access to AKS resource without vNet resource, then you are cannot reach to Kubernetes API (this is the meaning of private cluster)
  • with run-command preview feature turned on, when your client give you access to AKS resource, it allow you to access kubernetes API with limited kubectl command.
phillebaba commented 3 years ago

Is there any documentation for how this feature can be disabled in AKS?

haitch commented 3 years ago

CLI will be ready shortly, but you can use API to disable it for now:

https://github.com/Azure/azure-rest-api-specs/blob/191a721de644cc3f872f4fe9d676cf366083a106/specification/containerservice/resource-manager/Microsoft.ContainerService/stable/2021-08-01/managedClusters.json#L3901

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/containerservice/resource-manager/Microsoft.ContainerService/stable/2021-08-01/examples/ManagedClustersCreate_DisableRunCommand.json