kestra-io / kestra

:zap: Workflow Automation Platform. Orchestrate & Schedule code in any language, run anywhere, 500+ plugins. Alternative to Zapier, Rundeck, Camunda, Airflow...
https://kestra.io
Apache License 2.0
10.65k stars 914 forks source link

Azure storage fail to upload big files with IllegalStateException #5383

Closed loicmathieu closed 1 week ago

loicmathieu commented 1 week ago

Describe the issue

When trying to execute a flow with a FILE input using a large file (4MB for ex), it fails with an error java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread default-nioEventLoopGroup-1-3 with the Azure internal storage.

java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread default-nioEventLoopGroup-1-3
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
    at reactor.core.publisher.Mono.block(Mono.java:1779)
    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:146)
    at com.azure.storage.blob.BlobClient.uploadWithResponse(BlobClient.java:337)
    at com.azure.storage.blob.BlobClient.uploadWithResponse(BlobClient.java:298)
    at com.azure.storage.blob.BlobClient.upload(BlobClient.java:243)
    at io.kestra.storage.azure.AzureStorage.put(AzureStorage.java:199)
    at io.kestra.core.storages.StorageInterface.put(StorageInterface.java:75)
    at io.kestra.core.storages.StorageInterface.from(StorageInterface.java:96)
    at io.kestra.core.runners.FlowInputOutput.lambda$readData$2(FlowInputOutput.java:166)

Upstream issue TBD

Environment

loicmathieu commented 1 week ago

Upstream issue: https://github.com/Azure/azure-sdk-for-java/issues/42268