Azure storage : impossible to list queues

[zbendhiba]

It's impossible to iterate on the Azure PageIterable object with Camel Quarkus. So the usage of stream in this line is failing, even on JVM mode:

Example of error on JVM mode when we try the azure code on Quarkus runtime:

Caused by: com.azure.storage.queue.models.QueueStorageException: Status code 400, (empty body)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:370)
    at com.azure.core.http.rest.RestProxy.lambda$ensureExpectedStatus$5(RestProxy.java:410)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
    at reactor.core.publisher.Mono.subscribe(Mono.java:4361)
    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:82)
    at reactor.core.publisher.Operators.complete(Operators.java:137)
    at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:46)
    at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
    at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.java:189)
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:180)
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:284)
    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187)
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
    at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:160)
    at org.apache.camel.quarkus.support.azure.core.http.vertx.VertxHttpResponseHandler.handle(VertxHttpResponseHandler.java:52)
    at org.apache.camel.quarkus.support.azure.core.http.vertx.VertxHttpResponseHandler.handle(VertxHttpResponseHandler.java:29)
    at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:400)
    at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:387)
    at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:365)
    at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:332)
    at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:294)
    at io.vertx.ext.web.client.impl.HttpContext.lambda$null$8(HttpContext.java:550)
    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63)
    at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    ... 2 more
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        at reactor.core.publisher.Flux.blockLast(Flux.java:2644)
        at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94)
        at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.<init>(ContinuablePagedByItemIterable.java:50)
        at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:37)
        at com.azure.core.util.paging.ContinuablePagedIterable.iterator(ContinuablePagedIterable.java:106)

[zbendhiba]

Reproducer with resource class and test class

[jamesnetherton]

The list queues API request is invalid as there's no value for the include query param:

http://localhost:49242/devstoreaccount1?comp=list&include=

You need to set header QueueConstants.QUEUES_SEGMENT_OPTIONS with a simple value of new QueuesSegmentOptions() should be enough.

Arguably it's a bug in the Azure SDK or the Camel component that allows the empty segment options. So we could look to improve things here.

[zbendhiba]

@jamesnetherton this worked for you? It still failing for me.

I looked at the Standalone Java I've created last week with Azure SDK, the method with SegmentOption=null is working fine.

This is working in Standalone Java + Azure SDK :

List<QueueItem> list = queueServiceClient.listQueues(null, null, Context.NONE).stream().collect(Collectors.toList());
            for(QueueItem queue:list){
                // Output each queue name.
                System.out.println("************" + queue.getName());
            }

In camel quarkus, the only part that is failing is the stream on the result. If I debug the queueServiceClient.listQueues(null, null, Context.NONE), I can see on JVM mode that there is an object in the response.

I need to test with camel standalone, to check if it works fine with it.

BTW the official documentation points to this method https://github.com/Azure/azure-sdk-for-java/blob/8b6aadc2bede033373766e4f84ce331b68c2f19e/sdk/storage/azure-storage-queue/src/main/java/com/azure/storage/queue/QueueServiceClient.java#L229, that puts NULL

[zbendhiba]

Also, for some reason this is what I have with or without setting the QueueSegmentOptions http://localhost:49178/devstoreaccount1?comp=REDACTED&include=REDACTED

which seems to be different from your test

[jamesnetherton]

Ah sorry, seems I tricked myself into thinking it was working.

I also had options.setIncludeMetadata(true), which does make the test pass.

[zbendhiba]

@jamesnetherton This use case work, and could be a workaround.

I'll add the test in camel core project to check if it's a general issue.

[zbendhiba]

After many times testing, it turns out that is a pure Azurite bug. Using real Azure account fixes this issue.
Filled an Azurite issue : https://github.com/Azure/Azurite/issues/1426

[ppalaga]

Are we using the newest Azure SDK?

[zbendhiba]

@ppalaga we're not, but I did my standalone Java tests based on latest version of SDK. On camel-core main too, that uses latest version. And the behavior is just the same as the version used in CQ. There's no bug with SDK, only with Azurite response that misses some bits.