[BUG] StacklessClosedChannelException on Blob Downloads in Windows Azure Functions on Version 12.25.4

[SoulKa]

Describe the bug I am getting io.netty.channel.StacklessClosedChannelException exceptions when downloading blobs using downloadToFile() in version 12.25.4. The same code works using version 12.25.3. It is running in an Azure Function. The download link is valid and the blob storage is available. I can reproduce the error. Probably an issue due to the netty upgrade in version 12.25.4?

Exception or Stack Trace

Exception: io.netty.channel.StacklessClosedChannelException
Stack: reactor.core.Exceptions$ReactiveException: io.netty.channel.StacklessClosedChannelException
    at reactor.core.Exceptions.propagate(Exceptions.java:396)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:98)
    at reactor.core.publisher.Mono.block(Mono.java:1742)
    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:133)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1208)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1169)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:1078)
    < CENSORED USER CLASSES >
    at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
    at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:22)
    at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19)
    at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
    at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invoke(JavaFunctionBroker.java:133)
    at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:124)
    at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34)
    at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
    at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44)
    at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)
        ... 28 more
Caused by: io.netty.channel.StacklessClosedChannelException
    at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)
    Suppressed: io.netty.channel.StacklessClosedChannelException
        ... 1 more
    Suppressed: io.netty.channel.StacklessClosedChannelException
        ... 1 more
    Suppressed: io.netty.channel.StacklessClosedChannelException
        ... 1 more

To Reproduce Steps to reproduce the behavior:

1. Create an Azure Function that creates a blob client using new BlobClientBuilder().endpoint(blobSasUrl).buildClient() and a valid download URL to a blob with a valid SAS URL. Version 12.25.4 of the storage library has to be used.
2. Download an existing blob with client.downloadToFile(destinationPath, true)
3. Get the StacklessClosedChannelException

Code Snippet

var client = new BlobClientBuilder().endpoint(blobSasUrl).buildClient();
client.downloadToFile(destinationPath, true);

Expected behavior The blob gets successfully downloaded as it was in version 12.25.3

Setup (please complete the following information):

• OS: Windows
• IDE: IntelliJ Ultimate
• Library/Libraries: com.azure:azure-sdk-bom:1.2.23
  • com.azure:azure-core-http-netty:1.14.2
  • com.azure:azure-storage-blob:12.25.4
• Java version: 17
• App Server/Environment: Azure Function running on Runtime 4.33.1.22394 and App Service Plan EP2 and EP3, both getting the same issue
• Frameworks: None, only Azure Functions SDK

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [X] Bug Description Added
• [X] Repro Steps Added
• [X] Setup information Added

[github-actions[bot]]

@ibrahimrabab @ibrandes @seanmcc-msft

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[SoulKa]

I just ran the function locally on my mac. I could not reproduce the error there. It has to do with the Windows Function environment in combination with the new library...

[SoulKa]

Update: I noticed that the HTTP download of the blob simply does not progress. So the blob storage closes the TCP/HTTP connection after ~45s thus the exception. It seems that this is the origin of the issue. I do not know the internals of the Azure netty or blob storage download implementation, but some change in 12.25.4 results in the freezing HTTP call.

Please tell me if I can provide any useful information or tests.

[SoulKa]

Is someone working on this? Azure Functions seem like a common use case for blob downloads. It would be bad if this issue persists in the newer versions.

[ibrahimrabab]

Hi @SoulKa We are investigating this issue on our end and will get back to you on this thread as soon as we can with our findings. Thank you for your patience!

[SoulKa]

Thank you for investigating this issue! I can provide further details where needed

[ibrahimrabab]

Hi @SoulKa

Wanted to confirm, have you used either of these guides when trying to develop with Storage and Azure Functions?

https://learn.microsoft.com/en-us/azure/developer/java/sdk/troubleshooting-dependency-version-conflict#configure-azure-functions

https://learn.microsoft.com/en-us/azure/developer/java/sdk/troubleshooting-dependency-version-conflict#create-a-fat-jar

[SoulKa]

Hi @ibrahimrabab,

I have not used these guides yet. Are they relevant when I do not have a netty dependency in my project? These are my dependencies:

pom.xml

```xml com.microsoft.azure.functions azure-functions-java-library 3.1.0 com.azure azure-storage-blob org.junit.jupiter junit-jupiter 5.10.2 test org.mockito mockito-core 5.11.0 test org.mockito mockito-junit-jupiter 5.11.0 test lombok org.projectlombok provided 1.18.32 jackson-databind com.fasterxml.jackson.core 2.17.1 jackson-datatype-jsr310 com.fasterxml.jackson.datatype 2.17.1 annotations org.jetbrains provided 24.1.0 com.azure azure-sdk-bom 1.2.22 pom import ```

[ibrahimrabab]

@SoulKa While your POM doesn't explicitly include Netty, since it's a dependency of azure-storage-blob it will be included in the application. So, the guides shared are relevant.

Once you go through the guides, please let us know if you are still facing the issue, thank you!