stepanv
commented
2 years ago To work around this problem, one can instruct the reactive stream to cancel on a different thread. But that needs to be done after the particular thread executes MonoCollectList.cancel(). That is to declare it before the .collectList() statement.
var publish = Flux.using(
() -> driver.rxSession(SessionConfig.forDatabase(GraphDatabaseSettings.DEFAULT_DATABASE_NAME)),
session -> session.readTransaction(tx -> Flux.from(tx.run("call db.ping()").records())),
rxSession -> {
rxSession.close();
}
)
.cancelOn(Schedulers.parallel())
.collectList();
note that this wouldn't work as the parallel thread would get into a deadlock:
.collectList()
.cancelOn(Schedulers.parallel());but note that this work around works only thanks to the fact that other implementations of the Subscription interface do not use the same lock for onNext() and cancel().
injectives
We detected a deadlock in our production setup between 2 threads, each locking same locks but in the opposite order.
See the 2 callstacks from a thread dump we collected:
Here you can see that
Neo4jDriverIO-2-3is emitting data and while executing the on next chain it first locksBasicPullResponseHandlermonitor in the methodonRecord()and later it tries to lock theMonoCollectListSubscribermonitor in the methodonNext()default-nioEventLoopGroup-1-2which is running on an external event (in this case it's netty publishing a channel close which in turn makes Micronaut to cancel the reactive chain) and locks theMonoCollectListSubscriberwhile runningcancel()and tries to lockBasicPullResponseHandlerin thecancel()methodthe threads happen to end up in a deadlock.
Steps to reproduce
import java.time.Duration;
/**
You have as much time as you need to collect the thread dump. */ class Neo4jDeadlockTest {
@Test void neo4j_deadlock_withMonoList() { // Given Config config = Config.builder() .withoutEncryption() .withLogging(Logging.slf4j()) .withEventLoopThreads(5) .build();
} }
mainmethod tries to cancel the subscription via theblock(Duration.ofSeconds(5))mainthread and theNeo4jDriverthread got into a deadlockThis reproducer is just a demonstration of how one can reproduce the problem completely synthetically. By using the breakpoint you're causing the threads to interleave in the exact wrong order and ending in a deadlock.
We hit this deadlock while running a standard business logic in our production on AWS. (No breakpoints needed :) )
Expected behavior
The neo4j
BasicPullResponseHandlershould not get into a deadlock with Reactor primitives.