search

kubernetes-sigs / external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services
Apache License 2.0
7.7k stars 2.56k forks source link

No azure identity found for request clientID using azure-private-dns provider #4149

Open khauser opened 10 months ago

khauser commented 10 months ago

What happened:

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md#managed-identity-using-aks-kubelet-identity I tried these commands adapted to a private dns zone.

PRINCIPAL_ID=$(az aks show --subscription $CLUSTER_SUBSCRIPTION --resource-group $CLUSTER_GROUP --name $CLUSTERNAME \
  --query "identityProfile.kubeletidentity.objectId" --output tsv)
IDENTITY_CLIENT_ID=$(az aks show --subscription $CLUSTER_SUBSCRIPTION --resource-group $CLUSTER_GROUP --name $CLUSTERNAME \
  --query "identityProfile.kubeletidentity.clientId" --output tsv)

AZURE_DNS_ZONE="<example.com>" # DNS zone name like example.com or sub.example.com
AZURE_DNS_ZONE_RESOURCE_GROUP="<ours>" # resource group where DNS zone is hosted
AZURE_DNS_ZONE_SUBSCRIPTION="<ours>"

# fetch DNS id used to grant access to the kubelet identity
DNS_ID=$(az network private-dns zone show --name $AZURE_DNS_ZONE \
  --subscription $AZURE_DNS_ZONE_SUBSCRIPTION --resource-group $AZURE_DNS_ZONE_RESOURCE_GROUP --query "id" --output tsv)

az role assignment create --role "Private DNS Zone Contributor" --assignee $PRINCIPAL_ID --scope $DNS_ID # also fails with IDENTITY_CLIENT_ID

The last command fails with: If the assignee is an appId, make sure the corresponding service principal is created with 'az ad sp create --id

But I was able to create the role assignment manually through the Azure Portal UI.

But then also the external-dns pod gives: no azure identity found for request clientID

What you expected to happen:

Both the azure role assignment and the external-dns pod should find the AKS assigned managed identity.

How to reproduce it (as minimally and precisely as possible):

external-dns configuration file "helm/external-dns-values.yaml":

extraVolumeMounts:
  - name: azure-config-file
    mountPath: /etc/kubernetes
    readOnly: true
extraVolumes:
  - name: azure-config-file
    secret:
      secretName: azure-config-file
sources:
  - ingress
  - service

External-dns installation script:

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns
helm repo update

subscriptionId=$(az account show --subscription $AZURE_DNS_ZONE_SUBSCRIPTION --query "id" --output tsv)
tenantId=$(az account show --query "tenantId" --output tsv)

cat <<EOF >> azure.json
{
  "tenantId": "$tenantId",
  "subscriptionId": "$subscriptionId",
  "resourceGroup": "$AZURE_DNS_ZONE_RESOURCE_GROUP",
  "useManagedIdentityExtension": true,
  "userAssignedIdentityID": "<the kublet identity>"
}
EOF

namespace="external-dns"
if ! kubectl get namespace "$namespace" &> /dev/null; then
  kubectl create namespace "$namespace"
fi

secret_name="azure-config-file"
if kubectl get secret "$secret_name" -n "$namespace" &> /dev/null; then
  kubectl delete secret $secret_name -n "$namespace"
fi
kubectl create secret generic $secret_name -n external-dns --from-file azure.json

helm upgrade --install external-dns external-dns/external-dns --version 1.13.1 \
    --install \
    --wait \
    --namespace external-dns \
    --set provider=azure-private-dns \
    --set logLevel=debug \
    --values ./helm/external-dns-values.yaml \
    --create-namespace

Our aks.bicep with where the managed identity is created default wise:

param ppgId string
param aksName string
param aksVersion string
param aksVMSize string
param aksMaxScalingCount int
param vnetIPPrefix int

@secure()
param sshPublicKey string

var location = resourceGroup().location
var resourceGroupName = resourceGroup().name
var subscriptionId = subscription().subscriptionId

resource aksModule 'Microsoft.ContainerService/managedClusters@2023-07-01' = {
  name: aksName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  sku: {
    name: 'Base'
    tier: 'Standard'
  }
  properties: {
    kubernetesVersion: aksVersion
    dnsPrefix: aksName
    addonProfiles: {
      azureKeyvaultSecretsProvider: {
        enabled: true
        config:{
          enableSecretRotation: 'true'
        }
      }
    }
    agentPoolProfiles: [
      {
        name: 'agentpool'
        vmSize: 'Standard_DS2_v2' // just a small machine for the "unneeded" machine
        osDiskSizeGB: 30
        count: 1
        osType: 'Linux'
        mode: 'System'
        vnetSubnetID: resourceId('Microsoft.Network/virtualNetworks/subnets', '${aksName}-vnet', 'default')
        type: 'VirtualMachineScaleSets'
        orchestratorVersion: aksVersion
        enableNodePublicIP: false
        proximityPlacementGroupID: ppgId
        enableAutoScaling: true
        minCount: 1
        maxCount: 1
      }
      {
        name: 'agentpool2'
        vmSize: aksVMSize
        osDiskSizeGB: 128
        count: 0
        osType: 'Linux'
        mode: 'User'
        vnetSubnetID: resourceId('Microsoft.Network/virtualNetworks/subnets', '${aksName}-vnet', 'default')
        type: 'VirtualMachineScaleSets'
        orchestratorVersion: aksVersion
        enableNodePublicIP: false
        maxPods: 35
        enableAutoScaling: true
        minCount: 0
        maxCount: aksMaxScalingCount
      }
    ]
    autoScalerProfile: {
      'scale-down-utilization-threshold': '0.20'
    }
    networkProfile: {
      loadBalancerSku: 'standard'
      networkPlugin: 'azure'
      serviceCidr: '10.208.${vnetIPPrefix}.0/20'
      dnsServiceIP: '10.208.${vnetIPPrefix}.10'
      dockerBridgeCidr: '173.17.0.1/16'
    }
    linuxProfile: {
      adminUsername: 'localhorst'
      ssh: {
        publicKeys: [
          {
            keyData: sshPublicKey
          }
        ]
      }
    }
    enableRBAC: true
  }
  dependsOn: [
    aksName_vnet
  ]
}

resource aksName_vnet 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: '${aksName}-vnet'
  location: location
  properties: {
    subnets: [
      {
        name: 'default'
        id: '/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Network/virtualNetworks/${aksName}-vnet/subnets/default'
        properties: {
          addressPrefix: '10.208.${(vnetIPPrefix + 16)}.0/20'
          //10.208.16.0 - 10.208.16.255 prod
          //10.208.48.0 - 10.208.48.255 test
        }
      }
    ]
    addressSpace: {
      addressPrefixes: [
        '10.208.${vnetIPPrefix}.0/19'
        //10.208.0.0 - 10.208.31.255 prod
        //10.208.32.0 - 10.208.63.255 test
      ]
    }
  }
  tags: {}
}

output agentPoolIdentity object = aksModule.properties.identityProfile.kubeletidentity
output agentPoolIdentityObjectId string = aksModule.properties.identityProfile.kubeletidentity.objectId
output aksKeyVaultProviderIdentity object = aksModule.properties.addonProfiles.azureKeyvaultSecretsProvider.identity
output aksKeyVaultProviderIdentityObjectId string = aksModule.properties.addonProfiles.azureKeyvaultSecretsProvider.identity.objectId

Anything else we need to know?: Please ask if you need more information.

Environment: Azure AKS 1.27.3

khauser commented 10 months ago

I now also assigned "Reader" to the underlying resource group of the private dns zone but still external-dns is failing:

time="2024-01-05T09:31:50Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[ingress service] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:azure-private-dns GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[test.intershop.com] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareDNSRecordsPerPage:100 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxNameRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888 WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false}"
time="2024-01-05T09:31:50Z" level=info msg="Instantiating new Kubernetes client"
time="2024-01-05T09:31:50Z" level=debug msg="apiServerURL: "
time="2024-01-05T09:31:50Z" level=debug msg="kubeConfig: "
time="2024-01-05T09:31:50Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2024-01-05T09:31:50Z" level=info msg="Created Kubernetes client https://10.208.0.1:443"
time="2024-01-05T09:31:51Z" level=info msg="Using managed identity extension to retrieve access token for Azure API."
time="2024-01-05T09:31:51Z" level=debug msg="Retrieving Azure Private DNS zones for Resource Group 'plc-private-dns'"
time="2024-01-05T09:33:35Z" level=fatal msg="ManagedIdentityCredential authentication failed\nGET http://169.254.169.254/metadata/identity/oauth2/token\n--------------------------------------------------------------------------------\nRESPONSE 404 Not Found\n--------------------------------------------------------------------------------\nno azure identity found for request clientID <mine>\n\n--------------------------------------------------------------------------------\nTo troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#managed-id"
khauser commented 10 months ago

I tried it now with an own Service Principal and also with a workload identity. Both are working... except the kubelet identity.

PixelRobots commented 7 months ago

I am also seeing this issue.

bohdanbinahai commented 7 months ago

same issue

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

khauser commented 3 months ago

/remove-lifecycle rotten

Chandra2614 commented 2 months ago

any update on this?

Chandra2614 commented 2 months ago

I tried it now with an own Service Principal and also with a workload identity. Both are working... except the kubelet identity.

Mine is not working with workload identity, I am passing the client id of user assigned identity(which has federated credential) while running from Azure devops self hosted runners