Fail to load Azure AD Authentication configuration when AzureAD:xxx env var are mounted from KeyVault CSI SecretStore

[mleneveut]

Describe the bug I have mounted a CSI secret store from my KeyVault. It works for all attributes, except AzureAD:Domain, AzureAD:TenantId, AzureAD:Instance. When the AzureAD Authentication loads, it seems to use a file:///mnt/xxx/yyy instead of the KeyVaulkt secret value.

I got the error :

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: \u0027file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration\u0027.
at Microsoft.IdentityModel.Protocols.ConfigurationManager\u00601.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationHandler\u00601.AuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)

If I deactivate Swagger, I got the same error :

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: \u0027file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration\u0027.
at Microsoft.IdentityModel.Protocols.ConfigurationManager\u00601.GetConfigurationAsync(CancellationToken cancel)
 at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationHandler\u00601.AuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)

If instead of the mounted values I put hard coded values for AzureADDomain, AzureADTenantId, AzureAD__Instance, it works fine.

To Reproduce Steps to reproduce the behavior:

1. Mount a CSI Secret Store
2. Use an ASPNET 5.0 application with AzureAD Authentication enabled
3. Run the application in AKS 1.24.6

Expected behavior The AzureAD Authentication loads.

Manifest Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      serviceAccountName: workload-identity-dev-sa
      containers:
      - name: front
        image: xxx.azurecr.io/xxx:yyy
        ports:
        - containerPort: 8080
        env:
          - name: "AzureAd__ClientId"
            value: "8fe7d5bc-xxx-xxx-xxx-3d44be83cb20"
          - name: "AzureAd__Instance"
            value: "/mnt/xxx-kv/AzureAdInstance"
          - name: "AzureAd__Domain"
            value: "/mnt/xxx-kv/AzureAdDomain"
          - name: "AzureAd__TenantId"
            value: "/mnt/xxx-kv/AzureAdTenantId"
          - name: "InstallationServiceBusConnectionString"
            value: "/mnt/xxx-kv/installationCoreServiceBusConnectionString"
        volumeMounts:
        - name: xxxName
          mountPath: /mnt/xxx-kv
          readOnly: true
        imagePullPolicy: Always
      volumes:
        - name: xxxName
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: xxx-secret-provider

CSI Secret Store

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: xxx-secret-provider
  namespace: dev
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "xxx"
    keyvaultName: "xxx"
    objects:  |
      array:
        - |
          objectName: AzureAdInstance
          objectType: secret
          objectVersion: ""
        - |
          objectName: AzureAdDomain
          objectType: secret
          objectVersion: ""
        - |
          objectName: AzureAdTenantId
          objectType: secret
          objectVersion: ""
        - |
          objectName: installationCoreServiceBusConnectionString
          objectType: secret
          objectVersion: ""
    tenantId: "xxx"

Startup.cs

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            IdentityModelEventSource.ShowPII = true;

            app.UseRouting();

            app.UseAuthentication();
            app.UseAuthorization();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }

        public virtual void ConfigureServices(IServiceCollection services)
        {
            services.AddMicrosoftIdentityWebApiAuthentication(Configuration);
            services.Configure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme,
                options => {
                    options.RequireHttpsMetadata = false;
                }
            );

            services
                .AddControllers()
                .AddJsonOptions(opt => opt.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter()));

            var sliceAssembly = typeof(BackMeActivationTrackingController).Assembly;
            services.AddControllers()
                .PartManager.ApplicationParts.Add(new AssemblyPart(sliceAssembly));
        }

Environment (please complete the following information):

• Kubernetes version 1.24.6

[andyzhangx]

@aramase could you take a look? thanks.

[mleneveut]

@aramase Hi, could you have a try ?

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[olsenme]

@aramase?

[aramase]

I have mounted a CSI secret store from my KeyVault. It works for all attributes, except AzureAD:Domain, AzureAD:TenantId, AzureAD:Instance. When the AzureAD Authentication loads, it seems to use a file:///mnt/xxx/yyy instead of the KeyVaulkt secret value.

@mleneveut Could you clarify what the question is here? Is the CSI driver failing to mount the required secrets from AKV? If the CSI driver is mounting all the secrets as expected and you confirmed the values look fine in the mount, but the application is having issues consuming the secret, it might be good to check the code/reach out to the SDK folks.

[mleneveut]

@aramase Thanks for your response. The CSI is working fine for some secrets. I only have this problem with the Azure AD configuration.

Instead of giving the value of the secret to the configuration, it seems to give the path of the file containing the secret : the configuration setup constructs this URL : file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration

As shown in the first post, I don't give anything special to the Azure AD Configuration, just :

            services.AddMicrosoftIdentityWebApiAuthentication(Configuration);
            services.Configure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme,
                options => {
                    options.RequireHttpsMetadata = false;
                }
            );

If I set hard coded values in the Kubernetes yml for AzureAD__Instance and AzureAD_TenantID, it works fine.

Could you redirect me to the SDK folks if you think it is not in this github ? Thanks.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads