search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.33k stars 1.97k forks source link

[BUG] Blob storage: AuthorizationPermissionMismatch #37681

Open tpetillot opened 10 months ago

tpetillot commented 10 months ago

Describe the bug Java SDK blob operation failed with AuthorizationPermissionMismatch using AzureCliCredential and az login. But using CLI command works properly.

Exception or Stack Trace

[error] com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.
[error] If you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.
[error] Please remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.
[error] Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
[error] RequestId:c83039d6-b01e-0015-5ed8-17dadc000000
[error] Time:2023-11-15T15:26:13.0432670Z</Message></Error>"
[error]         at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
[error]         at com.azure.core.implementation.MethodHandleReflectiveInvoker.invokeWithArguments(MethodHandleReflectiveInvoker.java:35)
[error]         at com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:51)
[error]         at com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:356)
[error]         at com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:128)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
[error]         at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
[error]         at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2196)
[error]         at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2070)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
[error]         at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
[error]         at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
[error]         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
[error]         at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
[error]         at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
[error]         at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
[error]         at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
[error]         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
[error]         at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
[error]         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
[error]         at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292)
[error]         at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187)
[error]         at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
[error]         at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
[error]         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
[error]         at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
[error]         at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
[error]         at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
[error]         at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)
[error]         at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
[error]         at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)
[error]         at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)
[error]         at reactor.core.publisher.MonoUsing$MonoUsingSubscriber.onNext(MonoUsing.java:232)
[error]         at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
[error]         at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
[error]         at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:126)
[error]         at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
[error]         at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
[error]         at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:191)
[error]         at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
[error]         at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
[error]         at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)
[error]         at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
[error]         at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
[error]         at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
[error]         at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:437)
[error]         at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:491)
[error]         at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:753)
[error]         at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[error]         at com.azure.core.http.netty.implementation.AzureSdkHandler.channelRead(AzureSdkHandler.java:222)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[error]         at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
[error]         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
[error]         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
[error]         at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[error]         at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471)
[error]         at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1345)
[error]         at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)
[error]         at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
[error]         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
[error]         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[error]         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[error]         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[error]         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[error]         at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
[error]         at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
[error]         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
[error]         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
[error]         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[error]         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[error]         at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce Follow list the blobs in a container using java SDK guide.

Code Snippet

object AzureStorage {
  private val storageAccountName = "<account-name>"
  private val container = "<container-name>"

  def main(args: Array[String]): Unit = {
    val credentials = new AzureCliCredentialBuilder().build()
    val client = new BlobServiceClientBuilder()
      .endpoint(s"https://$storageAccountName.blob.core.windows.net/")
      .credential(credentials)
      .buildClient()

    val containerClient = client.getBlobContainerClient(container)
    println(s"Create completed: ${containerClient.createIfNotExists()}")

    containerClient.listBlobs().asScala.foreach(println)
  }
}

Expected behavior Works as well as CLI command: az storage blob list --account-name <account-name> --container-name <container-name>

Setup (please complete the following information):

ibrahimrabab commented 10 months ago

Hi @tpetillot Thank you for reaching out! I tried to reproduce this issue locally but did not run into any issues with authentication and was able to list blobs normally. I suspect there may be an issue with the account you're using to authenticate. Can you confirm that you're logged into the correct account in order to use the AzureCliCredentialBuilder?

Linking this here for more info: https://learn.microsoft.com/en-us/java/api/com.azure.identity.azureclicredentialbuilder?view=azure-java-stable

tpetillot commented 10 months ago

Hi @ibrahimrabab,

I can confirm the account used from the SDK is the one from the CLI. Some more context: it was able to create the container, but not list the blobs in it.

(For now, I use an app registration as a workaround that work properly).

ibrahimrabab commented 10 months ago

@billwert @g2vinay adding for visibility and any suggestions you folks may have for this issue.

ibrahimrabab commented 7 months ago

Hi @tpetillot Following up on this issue, is this still something you are experiencing?

github-actions[bot] commented 7 months ago

Hi @tpetillot. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

tpetillot commented 7 months ago

Hi @ibrahimrabab ,we've stuck to using our workaround ever since.

SabareeshGC commented 1 month ago

Facing same problem. Following this tutorial doesnt work https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-java?tabs=powershell%2Cmanaged-identity%2Croles-azure-portal%2Csign-in-azure-cli&pivots=blob-storage-quickstart-scratch

Following sample code breaks when trying to list blobs but works fine when listing the containers. I have confirmed that account that is logged to azure cli has appropriate access. I am using mac, intelij, spring. They are all using latest version of dependencies

package com.example.azurestorage;

import com.azure.core.credential.TokenCredential;
import com.azure.identity.AzureCliCredentialBuilder;
import com.azure.storage.blob.*;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.Objects;

@Configuration
public class AzureBlobConfig {
    @Bean
    public BlobServiceClient blobServiceClient() {

        TokenCredential credential =  new AzureCliCredentialBuilder().build();

        // Azure SDK client builders accept the credential as a parameter
        // TODO: Replace <storage-account-name> with your actual storage account name
        BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
                .endpoint("https://xxxxxx.blob.core.windows.net/")
                .credential(credential).buildClient();

        blobServiceClient.listBlobContainers()
                .forEach(item->{
            if(item != null)
                System.out.println(Objects.requireNonNull(item).getName());
        } );

        return blobServiceClient;
    }

    @Bean
    public BlobContainerClient userUploadsContainerClient(BlobServiceClient blobServiceClient
            , @Value("${app.azure.containers.userUploads:app-user-uploads}") String containerName) {

        BlobContainerClient client = blobServiceClient.getBlobContainerClient(containerName);

        client.createIfNotExists();

        client.listBlobs()               .forEach(item->{
            if(item != null)
                System.out.println(Objects.requireNonNull(item).getName());
        } );

        return client;
    }
}