search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.3k stars 1.96k forks source link

[BUG] exception is reported for azure-messaging-servicebus sdk component. ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception #13328

Closed MironAtHome closed 3 years ago

MironAtHome commented 4 years ago

Describe the bug Implementing async subscriber to receive messages, near copy from readme example, and running from custom thread from inside Microsoft Azure Workspace Spark Synapse, raises uncaught exception. The exception is raised at the time of setting up receiver. Further on, no messages can be received from the queue and a different exception is raised.

Exception or Stack Trace Exception #1: 20/07/16 02:10:42 ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler; at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:210) at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:41) at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:78) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510) at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510) at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at com.azure.core.amqp.implementation.ReactorSession.lambda$createConsumer$18(ReactorSession.java:402) at com.azure.core.amqp.implementation.handler.DispatchHandler.onTimerTask(DispatchHandler.java:36) at com.azure.core.amqp.implementation.ReactorDispatcher$WorkScheduler.run(ReactorDispatcher.java:130) at org.apache.qpid.proton.reactor.impl.SelectableImpl.readable(SelectableImpl.java:118) at org.apache.qpid.proton.reactor.impl.IOHandler.handleQuiesced(IOHandler.java:61) at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:390) at com.azure.core.amqp.implementation.handler.CustomIOHandler.onUnhandled(CustomIOHandler.java:39) at org.apache.qpid.proton.engine.BaseHandler.onReactorQuiesced(BaseHandler.java:87) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:206) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:292) at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Exception #2 Exception in thread "Thread-44" java.util.concurrent.RejectedExecutionException: ReactorDispatcher instance is closed. at com.azure.core.amqp.implementation.ReactorDispatcher.throwIfSchedulerError(ReactorDispatcher.java:93) at com.azure.core.amqp.implementation.ReactorDispatcher.invoke(ReactorDispatcher.java:68) at com.azure.core.amqp.implementation.ReactorReceiver.dispose(ReactorReceiver.java:169) at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.dispose(ServiceBusReactorReceiver.java:177) at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.checkAndSetTerminated(ServiceBusReceiveLinkProcessor.java:531) at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drainQueue(ServiceBusReceiveLinkProcessor.java:460) at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drain(ServiceBusReceiveLinkProcessor.java:450) at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.dispose(ServiceBusReceiveLinkProcessor.java:355) at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.close(ServiceBusAsyncConsumer.java:80) at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.close(ServiceBusReceiverAsyncClient.java:1145) at com.microsoft.azure.poc.StreamingPipeline.run(ExampleStream.scala:191) at java.lang.Thread.run(Thread.java:748)

To Reproduce Please find below is a sanitized version of IntelliJ IDEA project ZIP archive with code and build script prepared. Loading same from the context of Microsoft Azure Workspace Spark Synapse as Scala Job invariable runs into this exception Project.zip

Code Snippet please see ZIP archive above

Expected behavior Message received from Service Bus with no issues

Screenshots Not applicable

Setup (please complete the following information): Not applicable to the extent I can follow

Additional context Similar issue was reported for Microsoft Azure Event Hub Java API/SDK. Based on my understanding of developer dialogs logged in the issue, it appears that the trouble is somewhere between code as such and ability of the code to perform at high throughput. Please find is the issue link: https://github.com/Azure/azure-event-hubs-spark/issues/363

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

joshfree commented 4 years ago

@hemanttanwar PTAL

MironAtHome commented 4 years ago

Is PTAL stands for (P)lease (T)ake a (L)ook? I mean, I put together a very long and nice report. Can't you at least try to pretend, out of politeness, if not par for par, than at least a minimal amount of typing? So that I get an impression of commitment...

vinishgeorge commented 3 years ago

+1

ranjankumar03 commented 3 years ago

I am facing same issue .

ERROR 5860 --- [ctor-http-nio-2] reactor.core.publisher.Operators : Operator called default onErrorDropped

java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler; at com.azure.messaging.eventhubs.PartitionPumpManager.startPartitionPump(PartitionPumpManager.java:180) ~[azure-messaging-eventhubs-5.2.0.jar:na] at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$23(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_144] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_144] at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$24(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na] at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:137) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:794) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:560) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:540) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:426) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]

UmeshRajani commented 3 years ago

Facing same issue, I spent almost 5 hours to resolve and found here that it will be resolved in December milestone, hope so.

hemanttanwar commented 3 years ago

@ranjankumar03
But looking at the error, here is my analysis.

it seems like the setup in which ServiceBus is running is using older version of "projectreactor". The exception trace shows that it uses reactor-core-3.2.10.RELEASE.jar.

You can see that in this version Schedulers.boundedElastic() does not exists, Thus you are getting java.lang.NoSuchMethodError. 3.2.10.RELEASE API Reference: https://projectreactor.io/docs/core/3.2.10.RELEASE/api/reactor/core/scheduler/Schedulers.html

The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE".

We configure it here : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87

Can you ensure that this version is used ?

hemanttanwar commented 3 years ago

@MironAtHome I have not worked in "Microsoft Azure Workspace Spark Synapse", So I did not setup exactly same Synapse Workspace to replicate this. I will do some tutorials on how to set it up.

Thank you for sending Project.zip Appreciate it.

But I looked at your Stream_jar.xml and it looks like your setup is using "io.projectreactor:reactor-core:3.3.7.RELEASE:jar" ? Is it possible that your setup is using older version of "io.projectreacto:reactor-core" ? The cause for 'java.lang.NoSuchMethodError' is normally, if your setup is not using correct version of library.

The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE". We configure it here : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87

Also since your program is failing at ' exception is raised at the time of setting up receiver', it can not be performance related issue. To eliminate the performance issue: May be you can try reading just one message from queue and see if you can read it.

UmeshRajani commented 3 years ago

@hemanttanwar

Thanks Hemant, it's working perfect as with spring boot it was taking older version of reactor so explicitly added io.projectreacto:reactor-core:3.3.11.RELEASE in pom and works with Service Bus 7.0.0

hemanttanwar commented 3 years ago

@UmeshRajani Happy to see it resolved for you.

conniey commented 3 years ago

Glad this is resolved for you. NoSuchMethodErrors are often caused by mismatched dependencies. When I encounter these, I check the dependency tree. It'll show you all dependencies and how it decided to resolve the one causing your error.

https://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html

Ex: mvn dependency:tree -Dverbose -Dincludes=reactor*