AAD Pod Identity 1.7.1 does start on cluster deployed with kubenet and calico

[tvdvoorde]

Describe the bug

Not sure if this is a bug or confusing documentation

When I deploy an AKS cluster with

az aks create ... --network-policy calico --network-plugin kubenet

I can deploy the AAD Pod Identity 1.7.1, which should not be possible since the documentation now states it is not supported on kubenet because of the ARP spoofing vulnerability

However, in the code it checks for the --network-plugin flag on the kubelet, which, with the above combination of deployment flags is: --network-plugin=cni

Code snippet:

func IsKubenetCNI(kubeletConfig string) (bool, error) {
    var isKubenet bool
    // read the kubelet config
    bytes, err := ioutil.ReadFile(kubeletConfig)
    if err != nil {
        return isKubenet, err
    }
    return strings.Contains(string(bytes), "--network-plugin=kubenet"), nil
}

Steps To Reproduce

Deploy cluster with --network-policy calico --network-plugin kubenet

Deploy AAD Pod Identity 1.7.1

Expected behavior

NMI pod should not start

AAD Pod Identity version

1.17.1

Kubernetes version

1.19.3

Additional context

De Azure CLI appears to check differently - that checks only on the --network-plugin value on the CLI and blocks it either way, for the integrated AAD Pod Identity

It is blocked both with

az aks create ... --enable-pod-identity --network-plugin kubenet --network-policy calico

and with

az aks create ... --enable-pod-identity --network-plugin kubenet 

Results from commands:

>az aks create -g myResourceGroup -n myAKSCluster --enable-managed-identity --enable-pod-identity --network-plugin kubenet --network-policy calico
The behavior of this command has been altered by the following extension: aks-preview
BadRequestError: Operation failed with status: 'Bad Request'. Details: Network plugin kubenet is not supported to use with PodIdentity addon.

>az aks create -g myResourceGroup -n myAKSCluster --enable-managed-identity --enable-pod-identity --network-plugin kubenet --network-policy calico
The behavior of this command has been altered by the following extension: aks-preview
BadRequestError: Operation failed with status: 'Bad Request'. Details: Network plugin kubenet is not supported to use with PodIdentity addon.

[aramase]

@tvdvoorde AKS pod-managed identity is currently not supported with kubenet.

@bcho @miwithro can provide more details on this.

[tvdvoorde]

@tvdvoorde AKS pod-managed identity is currently not supported with kubenet.

Hi @aramase - thanks for the reply, I understand that. That is why I reported the issue.

When I deploy an AKS cluster with --network-plugin kubenet --network-policy calico I can still deploy it with the Yaml's or the Helm chart - because the code checks for the --network-plugin on the kubelet, and with the combination of kubenet+calico, the --network-plugin on the kubelet is defined as cni.

So is this a bug? Or is it still supported when combining kubenet with the calico network policy?

[aramase]

@tvdvoorde The managed add-on is handled by the AKS team. So I'll let @bcho and @miwithro comment on it.

[miwithro]

We are releasing an updated version that will allow the managed Add-on to run with kubenet.

[tvdvoorde]

@miwithro thanks for the info - do you know the timeline on the updated version?

[bcho]

Hi @tvdvoorde , the expected date would be around February 12th, we are working on enabling it in 0201 API and corresponding SDKs.

[b-odonoghue]

@bcho Just wanted to confirm that after this release around Feb 12th, this would allow the use of AAD pod identities for those using kubenet?

Moving the azureCNI is not really an option for us at this time, and there are some great integrations between AKS and other Azure services I would really like to take advantage of.

[bcho]

Hi @b-odonoghue , yes, we can enable this pod identity addon in cluster using kubenet after this release. We're in the progress of rollout.

[AvigdorLevy]

Hi @bcho , The support for AAD pod using kubenet is live? as trying to do so and it still fails with bad request..?

[bcho]

hey @AvigdorLevy , thanks for asking! We had rolled out the change to public regions, but this change requires to use 0201 API. We are still preparing the SDK and cli changes (related PR: https://github.com/Azure/azure-rest-api-specs/pull/12771)

[hieumoscow]

Hi @bcho, would the CVE-2021-1677 still be present in the addon and we would need to drop NET_RAW capability to mitigate it?

[miwithro]

@hieumoscow yes that is correct. For now you can follow this (https://github.com/ams0/opa-aad-pod-identity-kubenet) to assist. We will be adding an Azure Policy to assist with this soon.

[hieumoscow]

Thanks @miwithro, I have a requirement from Istio that NET_RAW needs to be enabled.

Anyway for us to satisfy both to run securely in Kubenet? Would dropping the capability on the main container only, not on Istio sidecar and deny exec to istio sidecars be sufficient to mitigate the CVE?

Thanks for your help

[miwithro]

Yes that works.

[srinathrangaramanujam]

Hey Guys. it seems IStio has released ISTIO CNI to mitigate the elevated caps. Have any of u guys had played with that :D. Not sure if this is a right place to ask. :)

[b-odonoghue]

@bcho i see your linked rest-api issue is now closed should I be able to enable or build a AKS cluster using kubenet CNI and also enable-pod-identity?

I still am getting errors that say kubenet does not support PodIdentity addon. I made sure to update my azure-cli and aks-preview extensions

[bcho]

@b-odonoghue hey, what's the version of aks-preview extensions are you using? We released as 0.5.3

[b-odonoghue]

@bcho Ahh I apologize I thought I had moved to that version but looks like I am still on 0.5.2.

➜  azure_tls az extension list --output table
Experimental    ExtensionType    Name         Path                                              Preview    Version
--------------  ---------------  -----------  ------------------------------------------------  ---------  ---------
False           whl              aks-preview  /Users/odonbr01/.azure/cliextensions/aks-preview  True       0.5.2

[aramase]

@bcho Is the rollout complete? Do we have updated docs on the commands to run for enabling pod identity in Kubenet?

[bcho]

@aramase yeah the rollout is complete, I am working with @miwithro to update the public doc

[aramase]

Thank you @bcho. We can close this issue once we have a reference to the public docs.

[aramase]

Closing this with https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity#using-kubenet-network-plugin-with-azure-active-directory-pod-managed-identities