search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.33k stars 1.97k forks source link

[BUG] AKS Workload Identity with Key Vault Spring property source is slow to start #29241

Closed scoof closed 1 year ago

scoof commented 2 years ago

Describe the bug When using AKS Workload Identity with Spring Cloud Key Vault PropertySource, startup takes ages. It seems like it takes 30s to resolve each property. The application works, but is just very slow to start up.

To Reproduce Steps to reproduce the behavior: Deploy the app in https://github.com/scoof/spring-keyvault-wi on an AKS cluster with Workload Identity enabled and using a Workload Identity that has access to your key vault - it will take 30s until the Spring banner is printed.

Expected behavior Faster startup than expected

Additional context Add any other context about the problem here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

joshfree commented 2 years ago

@chenrujun could you follow up with @scoof on this github issue?

chenrujun commented 2 years ago

@joshfree , sure, I'll investigate this issue.

chenrujun commented 2 years ago

Hi, @scoof .

Thanks for reaching out. I run https://github.com/scoof/spring-keyvault-wi in my localhost, it only take 8 seconds to start the SpringBootApplication. Here is the screenshot: image

I can image 3 potential reasons:

  1. There are too many secrets stored in your key vault. How many secrets did your key vault stored? If there are too many secrets stored in your key vault, you can use this property to narrow the scope:

    spring.cloud.azure.keyvault.secret.property-sources[0].secret-keys=my-secret

  2. It's caused by AKS Workload Identity. Which kind of workload identity are you using? You can refer to MS docs about workload identity to get more information about workload identity. If you are using managed identity, could you please try application? Here are the steps to use application: 2.1. Register an application, get client-id and client-secret. Refs: https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app 2.2. Configure access polity to the application. Refs: https://docs.microsoft.com/en-us/azure/key-vault/general/assign-access-policy?tabs=azure-portal 2.3. Add client-id and client-secret in the application.properties:

    spring.cloud.azure.keyvault.secret.property-source-enabled=true
spring.cloud.azure.keyvault.secret.property-sources[0].endpoint=https://test-vault.vault.azure.net/
spring.cloud.azure.keyvault.secret.property-sources[0].credential.client-id=
spring.cloud.azure.keyvault.secret.property-sources[0].credential.client-secret=
spring.cloud.azure.keyvault.secret.property-sources[0].profile.tenant-id

  3. It's caused by Azure resource: AKS or KeyVault. If it still show after using client-id and client-secret in AKS, please run the application in your localhost with client-id and client-secret. 3.1. If it still slow in your localhost, it may caused by the KeyVault resource, you can try another region when create KeyVault. 3.2. If it not slow in your localhost, it may cause by the AKS resource. Now I have no suggestion about this case. I'll continue investigate this case if you face this problem.

scoof commented 2 years ago

Hi @chenrujun - thanks for the quick reply.

  1. There are too many secrets stored in your key vault. How many secrets did your key vault stored? If there are too many secrets stored in your key vault, you can use this property to narrow the scope: 
    spring.cloud.azure.keyvault.secret.property-sources[0].secret-keys=my-secret

My sample vault only has one secret.

  1. It's caused by AKS Workload Identity. Which kind of workload identity are you using? You can refer to MS docs about workload identity to get more information about workload identity. If you are using managed identity, could you please try application? Here are the steps to use application:

It is definitely caused by AKS Workload Identity. I'm sorry if this wasn't clear from the initial issue. We're using the federated WI: https://azure.github.io/azure-workload-identity/docs/

Using local CLI auth works fine. If I extract the token file from the container and set AZURE_FEDERATED_TOKEN_FILE, AZURE_CLIENT_ID, and AZURE_TENANT_ID, I can reproduce locally.

scoof commented 2 years ago

I've just tried running it in a local container again, using: docker run -it --name kvwi -v /home/ap:/home/ap -e AZURE_TENANT_ID=[REDACTED] -e AZURE_FEDERATED_TOKEN_FILE=/home/ap/token -e AZURE_CLIENT_ID=[REDACTED] scoof/kvwi:1

and dumping the traffic.

Startup time was around 35s

The network pattern that I see is four HTTPS sessions: Connection to the vault (I assume that this is an initial unauthed connection). Folllowed by a connection to AAD (login.microsoftonline.com). Followed by a new Connection to the Key Vault. Then there's a 30s pause, and finally one more connection to AAD. So it seems like the app has all the secrets it needs, but is somehow stalling anyway.

chenrujun commented 2 years ago

@scoof

Thank you for your information. Seems that managed-identity cost a lot of time to get the access token.

What I did

I did this test in my localhost:

Test code

  public static void main(String[] args) {
    System.out.println("--- " + new Date() + " Start main.");
    SecretClient secretClient = new SecretClientBuilder()
            .vaultUrl("https://test.vault.azure.net/") // Update here
            .credential(new DefaultAzureCredentialBuilder().build())
            .buildClient();
    System.out.println("--- " + new Date() + " Build secretClient.");
    String secretValue = secretClient.getSecret("my-secret").getValue();
    System.out.println("--- " + new Date() + " Got secretValue.");
    System.out.println("--- " + new Date() + " secretValue = " + secretValue);
  }

Filter related log. (If you are using Intellij, you can filter log by grep console plugin)

Log 1: System out related log. (grep by ---)

--- Wed Jun 08 16:42:53 CST 2022 Start main.
--- Wed Jun 08 16:42:54 CST 2022 Build secretClient.
--- Wed Jun 08 16:43:00 CST 2022 Got secretValue.
--- Wed Jun 08 16:43:00 CST 2022 secretValue = my-secret-value

It shows that it only cost 7 seconds to build secretClient and get secretValue.

Log 2: Azure Identity related log. (grep by Azure Identity =>)

16:42:53.860 [main] DEBUG com.azure.identity.EnvironmentCredential - Azure Identity => Found the following environment variables: 
16:42:53.866 [main] DEBUG com.azure.identity.EnvironmentCredential - Azure Identity => ERROR in EnvironmentCredential: Missing required environment variable AZURE_CLIENT_ID
16:42:54.424 [main] DEBUG com.azure.identity.ManagedIdentityCredential - Azure Identity => Found the following environment variables: 
16:42:54.425 [main] DEBUG com.azure.identity.SharedTokenCacheCredential - Azure Identity => Found the following environment variables: 
16:42:57.530 [reactor-http-nio-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential EnvironmentCredential is unavailable.
16:42:57.557 [reactor-http-nio-1] DEBUG com.azure.identity.ManagedIdentityCredential - Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, Network is unreachable: no further information.
16:42:57.557 [reactor-http-nio-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential ManagedIdentityCredential is unavailable.
16:42:57.686 [ForkJoinPool.commonPool-worker-1] DEBUG com.azure.identity.SharedTokenCacheCredential - Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
16:42:57.686 [ForkJoinPool.commonPool-worker-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential SharedTokenCacheCredential is unavailable.
16:42:57.690 [ForkJoinPool.commonPool-worker-1] DEBUG com.azure.identity.IntelliJCredential - Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: IntelliJ Authentication not available. Please log in with Azure Tools for IntelliJ plugin in the IDE.
16:42:57.690 [ForkJoinPool.commonPool-worker-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential IntelliJCredential is unavailable.
16:42:57.847 [ForkJoinPool.commonPool-worker-1] DEBUG com.azure.identity.VisualStudioCodeCredential - Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: Failed to read Vs Code credentials from Windows Credential API.
16:42:57.848 [ForkJoinPool.commonPool-worker-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential VisualStudioCodeCredential is unavailable.
16:42:59.698 [ForkJoinPool.commonPool-worker-1] INFO com.azure.identity.AzureCliCredential - Azure Identity => getToken() result for scopes [https://vault.azure.net/.default]: SUCCESS
16:42:59.698 [ForkJoinPool.commonPool-worker-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential AzureCliCredential returns a token

It shows that it's using AzureCliCredential to get access token, and it didn't cost too much time.

My request

If I extract the token file from the container

I don't know how to extract the token file from the container, could you please run this code in your localhost? So you can get more information why it cost so much time to get access token.

scoof commented 2 years ago

@chenrujun

Your sample code does not seem to be hit by any delay.

log 1:

--- Wed Jun 08 11:40:27 CEST 2022 Start main.
--- Wed Jun 08 11:40:28 CEST 2022 Build secretClient.
--- Wed Jun 08 11:40:29 CEST 2022 Got secretValue.
--- Wed Jun 08 11:40:29 CEST 2022 secretValue = test-secret

log 2:

11:40:28.130 [main] DEBUG com.azure.identity.EnvironmentCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
11:40:28.132 [main] DEBUG com.azure.identity.EnvironmentCredential - Azure Identity => ERROR in EnvironmentCredential: Failed to create a ClientSecretCredential or ClientCertificateCredential. Missing required environment variable either AZURE_CLIENT_SECRET or AZURE_CLIENT_CERTIFICATE_PATH
11:40:28.133 [main] DEBUG com.azure.identity.EnvironmentCredential - Azure Identity => ERROR in EnvironmentCredential: Failed to determine an authentication scheme based on the available environment variables. Please specify AZURE_TENANT_ID and AZURE_CLIENT_SECRET to authenticate through a ClientSecretCredential; AZURE_TENANT_ID and AZURE_CLIENT_CERTIFICATE_PATH to authenticate through a ClientCertificateCredential; or AZURE_USERNAME and AZURE_PASSWORD to authenticate through a UserPasswordCredential.
11:40:28.268 [main] DEBUG com.azure.identity.ManagedIdentityCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
11:40:28.269 [main] DEBUG com.azure.identity.SharedTokenCacheCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID
11:40:28.976 [reactor-http-epoll-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential EnvironmentCredential is unavailable.
11:40:29.228 [reactor-http-epoll-1] INFO com.azure.identity.ManagedIdentityCredential - Azure Identity => Managed Identity environment: AZURE AKS TOKEN EXCHANGE
11:40:29.229 [reactor-http-epoll-1] INFO com.azure.identity.ManagedIdentityCredential - Azure Identity => getToken() result for scopes [https://vault.azure.net/.default]: SUCCESS
11:40:29.229 [reactor-http-epoll-1] INFO com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential ManagedIdentityCredential returns a token

To extract the token, you can just copy the file that the AZURE_FEDERATED_TOKEN_FILE env var points to.

scoof commented 2 years ago

@chenrujun Just to be clear, we're only seeing the delay when using PropertySource, not when interacting with the SecretClient directly.

scoof commented 2 years ago

@chenrujun do you need any more from me to get wiser?

chenrujun commented 2 years ago

@scoof

Sorry that I'm busy with other tasks recently, I didn't reproduce your problem. Is it urgent? I think 30 seconds delay is not a big problem.

If you want to investigate by yourself, here is my suggestion:

  1. Copy KeyVaultEnvironmentPostProcessor into your sample project, rename to XxxKeyVaultEnvironmentPostProcessor change the package name, and add more logs to give information about how much time each step consumed.
  2. Enable XxxKeyVaultEnvironmentPostProcessor by adding it to your sample project's spring.factories, just like this.
  3. Run your sample application, then you can get the information about time-consuming by the logs you added in step 1.
scoof commented 2 years ago

The delay is in the KeyVaultOperation ctor called here: https://github.com/Azure/azure-sdk-for-java/blob/spring-cloud-azure_4.2.0/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/keyvault/environment/KeyVaultEnvironmentPostProcessor.java#L120

Still digging

saragluna commented 2 years ago

Hey @g2vinay, does the azure-identity support AKS workload identity? @scoof is the workload identity here the workload identity federation for AKS?

scoof commented 2 years ago

@saragluna AKS Workload identity works and is part of the ManagedIdentityCredential: https://github.com/Azure/azure-sdk-for-java/blob/spring-cloud-azure_4.2.0/sdk/identity/azure-identity/src/main/java/com/azure/identity/ManagedIdentityCredential.java#L31

And that is indeed what we're using.

JFTR: It works. It's just slow.

scoof commented 2 years ago

The delay occurs in the generated http client called here: https://github.com/Azure/azure-sdk-for-java/blob/spring-cloud-azure_4.2.0/sdk/keyvault/azure-security-keyvault-secrets/src/main/java/com/azure/security/keyvault/secrets/SecretClient.java#L229

stliu commented 2 years ago

since the environment post processor is working on the environment prepare stage, so the spring boot log system is not ready, we can't see the log that causing it slow.

@scoof could you try build a secretClient in a pojo (e.g. ApplicationRunner) and enable the finest log level so we can see why it is slow

scoof commented 2 years ago

@stliu I think we already did what you're suggesting: https://github.com/Azure/azure-sdk-for-java/issues/29241#issuecomment-1149737762 - but somehow that didn't replicate the issue

scoof commented 2 years ago

Could this be because the Reactive threads have not been started yet?

saragluna commented 2 years ago

Can that explain why it only happens with workload identity?

scoof commented 2 years ago

I would expect it to happen with the environment auth too, since that would also have to send a request to AAD to get a token for the vault.

scoof commented 2 years ago

I've tried enabling OpenTelemetry tracing, and I am getting these results: First it tries to access the vault without auth: image

Then it does a token exchange: image

And then comes the weird part, where it accesses the vault with auth, but somehow pauses for 30s before sending the actual request: image

scoof commented 2 years ago

One more observation: The second request for the key vault reuses the TCP connection, so maybe this is caused by the pool not releasing the connection

scoof commented 2 years ago

It is related to connection reuse. If I replace this line with a static list of keys, the problem goes away: https://github.com/Azure/azure-sdk-for-java/blob/8f1158c74d3dd41b8772534dae4e18a535709649/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/keyvault/environment/KeyVaultEnvironmentPostProcessor.java#L123

saragluna commented 2 years ago

So if you copy the same code out of the environment post processor, it will reproduce the issue right?

scoof commented 2 years ago

Yes. Here's a non-Spring reproducer using refreshProperties() code from KeyVaultOperation.java:

package com.example.springkeyvaultwi;

import com.azure.core.http.rest.PagedResponse;
import com.azure.core.util.paging.ContinuablePagedIterable;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.security.keyvault.secrets.SecretClient;
import com.azure.security.keyvault.secrets.SecretClientBuilder;
import com.azure.security.keyvault.secrets.models.KeyVaultSecret;
import com.azure.security.keyvault.secrets.models.SecretProperties;
import java.util.Date;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.lang.NonNull;

public class SpringKeyvaultWiApplication {

  private static String toKeyVaultSecretName(@NonNull String property) {
    final boolean caseSensitive = false;
    if (!caseSensitive) {
      if (property.matches("[a-z0-9A-Z-]+")) {
        return property.toLowerCase(Locale.US);
      } else if (property.matches("[A-Z0-9_]+")) {
        return property.toLowerCase(Locale.US).replace("_", "-");
      } else {
        return property.toLowerCase(Locale.US)
            .replace("-", "") // my-project -> myproject
            .replace("_", "") // my_project -> myproject
            .replace(".", "-"); // acme.myproject -> acme-myproject
      }
    } else {
      return property;
    }
  }

  public static void main(String[] args) {
    final Logger logger = LoggerFactory.getLogger("getClass");

    System.out.println("--- " + new Date() + " Start main.");
    SecretClient secretClient = new SecretClientBuilder()
        .vaultUrl("https://apl-oidc-test.vault.azure.net/") // Update here
        .credential(new DefaultAzureCredentialBuilder().build())
        .buildClient();
    System.out.println("--- " + new Date() + " Build secretClient.");
    //Buggy code from KeyVaultOperation. The bug seems to be that the same SecretClient is used for getting the value while still iterating over the keys in the Key Vault
    final Map<Object, String> properties = Optional.of(secretClient)
        .map(SecretClient::listPropertiesOfSecrets)
        .map(ContinuablePagedIterable::iterableByPage)
        .map(i -> StreamSupport.stream(i.spliterator(), false))
        .orElseGet(Stream::empty)
        .map(PagedResponse::getElements)
        .flatMap(i -> StreamSupport.stream(i.spliterator(), false))
        .filter(SecretProperties::isEnabled)
        .map(p -> secretClient.getSecret(p.getName(), p.getVersion()))
        .filter(Objects::nonNull)
        .collect(Collectors.toMap(
            s -> toKeyVaultSecretName(s.getName()),
            KeyVaultSecret::getValue
        ));
    for (Map.Entry e: properties.entrySet()) {
      System.out.println("--- " + new Date() + " Got secretValue for key " + e.getKey());
      System.out.println("--- " + new Date() + " secretValue = " + e.getValue());
    }
  }

}

The culprit seems to be that it's fetching each value while still paging through the keys.

saragluna commented 2 years ago

Thanks! And this still only happens with workload identity right?

scoof commented 2 years ago

It did not happen when using az authentication or IMDS at least. I haven't tested others.

saragluna commented 2 years ago

Thanks for all the digging and analysis, it helps a lot!

scoof commented 2 years ago

So a workaround for those interested is to explicitly list the keys you want in your application.properties

saragluna commented 2 years ago

Yes. Here's a non-Spring reproducer using refreshProperties() code from KeyVaultOperation.java:

package com.example.springkeyvaultwi;

import com.azure.core.http.rest.PagedResponse;
import com.azure.core.util.paging.ContinuablePagedIterable;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.security.keyvault.secrets.SecretClient;
import com.azure.security.keyvault.secrets.SecretClientBuilder;
import com.azure.security.keyvault.secrets.models.KeyVaultSecret;
import com.azure.security.keyvault.secrets.models.SecretProperties;
import java.util.Date;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.lang.NonNull;

public class SpringKeyvaultWiApplication {

  private static String toKeyVaultSecretName(@NonNull String property) {
    final boolean caseSensitive = false;
    if (!caseSensitive) {
      if (property.matches("[a-z0-9A-Z-]+")) {
        return property.toLowerCase(Locale.US);
      } else if (property.matches("[A-Z0-9_]+")) {
        return property.toLowerCase(Locale.US).replace("_", "-");
      } else {
        return property.toLowerCase(Locale.US)
            .replace("-", "") // my-project -> myproject
            .replace("_", "") // my_project -> myproject
            .replace(".", "-"); // acme.myproject -> acme-myproject
      }
    } else {
      return property;
    }
  }

  public static void main(String[] args) {
    final Logger logger = LoggerFactory.getLogger("getClass");

    System.out.println("--- " + new Date() + " Start main.");
    SecretClient secretClient = new SecretClientBuilder()
        .vaultUrl("https://apl-oidc-test.vault.azure.net/") // Update here
        .credential(new DefaultAzureCredentialBuilder().build())
        .buildClient();
    System.out.println("--- " + new Date() + " Build secretClient.");
    //Buggy code from KeyVaultOperation. The bug seems to be that the same SecretClient is used for getting the value while still iterating over the keys in the Key Vault
    final Map<Object, String> properties = Optional.of(secretClient)
        .map(SecretClient::listPropertiesOfSecrets)
        .map(ContinuablePagedIterable::iterableByPage)
        .map(i -> StreamSupport.stream(i.spliterator(), false))
        .orElseGet(Stream::empty)
        .map(PagedResponse::getElements)
        .flatMap(i -> StreamSupport.stream(i.spliterator(), false))
        .filter(SecretProperties::isEnabled)
        .map(p -> secretClient.getSecret(p.getName(), p.getVersion()))
        .filter(Objects::nonNull)
        .collect(Collectors.toMap(
            s -> toKeyVaultSecretName(s.getName()),
            KeyVaultSecret::getValue
        ));
    for (Map.Entry e: properties.entrySet()) {
      System.out.println("--- " + new Date() + " Got secretValue for key " + e.getKey());
      System.out.println("--- " + new Date() + " secretValue = " + e.getValue());
    }
  }

}

The culprit seems to be that it's fetching each value while still paging through the keys.

Hi @g2vinay, seems like this is not a Spring issue. Could you help take a look?

stliu commented 1 year ago

@g2vinay @joshfree would you take a look and get back to the customer? thanks

scoof commented 1 year ago

It seems like the code is still the same: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/keyvault/environment/KeyVaultOperation.java#L135

stliu commented 1 year ago

It seems like the code is still the same: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/keyvault/environment/KeyVaultOperation.java#L135

@scoof yep, the code in spring auto configure is not changed since the root cause is the underlying KeyVault SDK, so the fix should happen there

scoof commented 1 year ago

It seems like the code is still the same: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/spring/spring-cloud-azure-autoconfigure/src/main/java/com/azure/spring/cloud/autoconfigure/keyvault/environment/KeyVaultOperation.java#L135

@scoof yep, the code in spring auto configure is not changed since the root cause is the underlying KeyVault SDK, so the fix should happen there

Is there anything we can do to expedite a fix? This issue makes it quite cumbersome for our devs to use KeyVault.

joshfree commented 1 year ago

Is there anything we can do to expedite a fix? This issue makes it quite cumbersome for our devs to use KeyVault.

Hi @scoof, in general we ship updates on a monthly cadence. If your issue requires immediate assistance the SUPPORT.md in the repo has further instructions: https://github.com/Azure/azure-sdk-for-java/blob/main/SUPPORT.md#azure-support-tickets

scoof commented 1 year ago

Is there anything we can do to expedite a fix? This issue makes it quite cumbersome for our devs to use KeyVault.

Hi @scoof, in general we ship updates on a monthly cadence.

But this ticket and reproducer are now several months old?

g2vinay commented 1 year ago

@scoof I am investigating this issue and will update you here with my findings and next steps soon.

g2vinay commented 1 year ago

@scoof

We shipped a fix related to this issue, in azure-identity:1.7.2.

<dependency>
  <groupId>com.azure</groupId>
  <artifactId>azure-identity</artifactId>
  <version>1.7.2</version>
</dependency>

Try it out, and let us know if you still face the issue.

saragluna commented 1 year ago

https://github.com/Azure/azure-sdk-for-java/pull/32306

scoof commented 1 year ago

This fixes the issue, thanks.