search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.36k stars 2k forks source link

[BUG] Authentication to Key Vault fails using ManagedIdentityCredentialBuilder with user-assigned managed identity #30043

Closed juergenbr closed 8 months ago

juergenbr commented 2 years ago

Describe the bug ManagedIdentityCredentialBuilder with User assigned managed identity client id causes the following error: 2022-07-20 05:40:55.983 ERROR 42 --- [ parallel-1] c.a.identity.ManagedIdentityCredential : Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: Server returned HTTP response code: 400 for URL: http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028

Exception or Stack Trace

... 3 more
at   java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at   reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
Caused by: java.io.IOException: Server   returned HTTP response code: 400 for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
at   java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
at   reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at   reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:34)
at   reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:53)
at   reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at   java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
... 30 more
at   com.azure.identity.implementation.IdentityClient.lambda$authenticateToManagedIdentityEndpoint$55(IdentityClient.java:1322)
at   reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at   java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at   reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
at   reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionOtherSubscriber.onNext(FluxDelaySubscription.java:131)
at   reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:57)
at   reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
at   reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
Executed 'Functions.validate' (Succeeded,   Id=c83c2ff9-f9b8-4714-86d4-2cfa567a32d2, Duration=90126ms)
Function "validate" (Id:   c83c2ff9-f9b8-4714-86d4-2cfa567a32d2) invoked by Java Worker
at   com.dynatrace.ace.heimdallazure.services.KeyVaultService.getKeyVaultSecret(KeyVaultService.java:47)
Suppressed: java.lang.Exception: #block   terminated with an error
... 26 more
at   com.azure.security.keyvault.secrets.SecretClient.getSecret(SecretClient.java:199)
at   reactor.core.Exceptions.propagate(Exceptions.java:392)
at   reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
at   com.azure.security.keyvault.secrets.SecretClient.getSecretWithResponse(SecretClient.java:229)
at   reactor.core.publisher.Mono.block(Mono.java:1707)
at   com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)
at   java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at   com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)
Caused by:   reactor.core.Exceptions$ReactiveException: java.io.IOException: Server   returned HTTP response code: 400 for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
at   java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at   java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at   com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)
at   java.base/java.lang.Thread.run(Thread.java:829)
at   com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
at   java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at   com.dynatrace.ace.heimdallazure.functions.validate.HeimdallAzureValidationFunction.initFunction(HeimdallAzureValidationFunction.java:70)
at   org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:708)
at   com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
at   java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at   org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunctionAndEnrichResultIfNecessary(SimpleFunctionRegistry.java:897)
at   org.springframework.cloud.function.adapter.azure.FunctionInvoker.handleRequest(FunctionInvoker.java:129)
at   com.dynatrace.ace.heimdallazure.functions.validate.HeimdallAzureValidationFunction.apply(HeimdallAzureValidationFunction.java:25)
at   com.dynatrace.ace.heimdallazure.functions.validate.HeimdallAzureValidationFunction.apply(HeimdallAzureValidationFunction.java:49)
at   com.dynatrace.ace.heimdallazure.functions.validate.HeimdallAzureValidationFunctionHandler.execute(HeimdallAzureValidationFunctionHandler.java:25)
at   java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at   org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
at   com.microsoft.azure.functions.worker.broker.EnhancedJavaMethodExecutorImpl.execute(EnhancedJavaMethodExecutorImpl.java:55)
at   java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native   Method)
at   com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:61)
at   org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunction(SimpleFunctionRegistry.java:853)
at   java.base/java.lang.reflect.Method.invoke(Method.java:566)
java.lang.RuntimeException: Error getting   secret from Key vault: java.io.IOException: Server returned HTTP response   code: 400 for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
at   com.dynatrace.ace.heimdallazure.services.KeyVaultService.getKeyVaultSecret(KeyVaultService.java:51)
Server returned HTTP response code: 400   for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:46:02.108  WARN 42 --- [     parallel-1]   c.a.s.k.secrets.SecretAsyncClient          : Failed to get secret - heimdall-azure-sp-clientid
2022-07-20 06:46:02.109 ERROR 42 ---   [pool-2-thread-5] c.d.a.h.SimpleExecutionContext           : Exception running Heimdall API   for subscription e017d26e-76aa-48cd-befc-5f95ed75b761
at   com.dynatrace.ace.heimdallazure.functions.validate.HeimdallAzureValidationFunction.createGraphServiceClient(HeimdallAzureValidationFunction.java:152)
2022-07-20 06:46:02.104  INFO 42 --- [     parallel-1]   com.azure.core.http.policy.RetryPolicy     : Retry attempts have been exhausted after 3 attempts.
2022-07-20 06:46:02.104 ERROR 42 ---   [     parallel-1]   c.a.identity.ManagedIdentityCredential     : Azure Identity => ERROR in getToken() call for scopes   [https://vault.azure.net/.default]: Server returned HTTP response code: 400   for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:46:02.104 ERROR 42 ---   [     parallel-1]   c.a.c.implementation.AccessTokenCache      : Failed to acquire a new access token.
2022-07-20 06:45:32.088 ERROR 42 ---   [     parallel-1]   c.a.c.implementation.AccessTokenCache      : Failed to acquire a new access token.
2022-07-20 06:45:32.086 ERROR 42 ---   [     parallel-1]   c.a.identity.ManagedIdentityCredential     : Azure Identity => ERROR in getToken() call for scopes   [https://vault.azure.net/.default]: Server returned HTTP response code: 400   for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:45:02.069 ERROR 42 ---   [     parallel-1]   c.a.identity.ManagedIdentityCredential     : Azure Identity => ERROR in getToken() call for scopes   [https://vault.azure.net/.default]: Server returned HTTP response code: 400   for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:45:02.069 ERROR 42 ---   [     parallel-1]   c.a.c.implementation.AccessTokenCache      : Failed to acquire a new access token.
2022-07-20 06:44:32.042 ERROR 42 ---   [pool-2-thread-5] c.a.identity.ManagedIdentityCredential   : Azure Identity => ERROR in getToken()   call for scopes [https://vault.azure.net/.default]: Server returned HTTP response   code: 400 for URL:   http://169.254.129.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01&client_id=a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:44:32.042 ERROR 42 ---   [pool-2-thread-5] c.a.c.implementation.AccessTokenCache    : Failed to acquire a new access token.
2022-07-20 06:44:32.024  INFO 42 --- [pool-2-thread-5]   c.d.a.h.SimpleExecutionContext             : Retrieving secret heimdall-azure-sp-clientid
2022-07-20 06:44:32.022  INFO 42 --- [pool-2-thread-5]   c.d.a.h.SimpleExecutionContext             : Found client id a81093e4-baeb-424d-b3ee-62dd7fcf5028
2022-07-20 06:44:32.020  INFO 42 --- [pool-2-thread-5]   c.d.a.h.SimpleExecutionContext             : Creating KeyVaultService
2022-07-20 06:44:32.020  INFO 42 --- [pool-2-thread-5]   c.d.a.h.SimpleExecutionContext             : Creating Key Vault client for vault dt-heimdall-azure-kv
Processing Heimdall Azure deployment   request
2022-07-20 06:44:32.014  INFO 42 --- [pool-2-thread-5]   o.s.c.function.utils.FunctionClassUtils    : Main class: class   com.dynatrace.ace.heimdallazure.HeimdallAzureApplication
Executing 'Functions.validate'   (Reason='This function was programmatically called via the host APIs.',   Id=c83c2ff9-f9b8-4714-86d4-2cfa567a32d2)

To Reproduce 1) Create Azure Function with VNet Integration and Private Endpoint 2) Add a user-assigned managed identity 3) Deploy function code that uses the Key Vault SDK and uses Managed Identity authentication 4) Call function and trigger execution of below code snipet

Code Snippet

private void getSecretClient() {
        String keyVaultName = System.getenv("KEYVAULTNAME");
        logger.info("Creating Key Vault client for vault " + keyVaultName);
        String keyVaultUri = "https://" + keyVaultName + ".vault.azure.net";
        this.secretClient = new SecretClientBuilder()
                .vaultUrl(keyVaultUri)
                .credential(buildCredentials())
                .buildClient();
    }

    public TokenCredential buildCredentials(){
        String clientId = System.getenv("MANAGED_IDENTITY_CLIENT_ID");
        logger.info("Found client id " + clientId);
        return new ManagedIdentityCredentialBuilder()
            .clientId(clientId)
            .build();
    }

    public String getKeyVaultSecret(String secretName) {
        try {
            logger.info("Retrieving secret " + secretName);
            KeyVaultSecret retrievedSecret = secretClient.getSecret(secretName);
            logger.info("Secret value: " + retrievedSecret);
            return retrievedSecret.getValue();
        } catch (Exception e) {
            throw new RuntimeException("Error getting secret from Key vault: " + e.getMessage(), e);
        }
    }

Expected behavior A successful call to the Key Vault to retrieve the secret value.

Screenshots Not applicable

Setup (please complete the following information):

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

Additional context Add any other context about the problem here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

joshfree commented 2 years ago

@g2vinay could you please take a look at this issue @juergenbr has filed? /cc @billwert

kalyanrn commented 2 years ago

I am also seeing this issue when I am using the Spring Boot 2.7.2 ,spring-cloud-azure-dependencies - 4.3.0, Java 11.

I am using Azure role-based access control and assigned the Key Vault Administrator role to the System Managed Identity (created from Webapp). I am using Object (principal) ID for the client-id.

I am seeing this issue both in User Managed and System Managed Identity

spring.cloud.azure.keyvault.secret.property-source-enabled=true spring.cloud.azure.keyvault.secret.property-sources[0].credential.client-id=XXXXX spring.cloud.azure.keyvault.secret.property-sources[0].credential.managed-identity-enabled=true spring.cloud.azure.keyvault.secret.property-sources[0].profile.tenant-id=XXXXXX spring.cloud.azure.keyvault.secret.property-sources[0].endpoint=XXXXX

moarychan commented 2 years ago

Hi @kalyanrn , thanks for using the Spring Cloud Azure Starter Key Vault Secret 4.3.0!

You can follow this just updated doc https://docs.microsoft.com/azure/spring-apps/tutorial-managed-identities-key-vault

g2vinay commented 1 year ago

@juergenbr

Can you try out the latest identity release v.1.9.1 and check if the issue still persists ?

g2vinay commented 8 months ago

Let us know if you still need any help/assistance on this.