Use own Executor for client connection loop to avoid deadlocks.

[dpasek-senacor]

Testing has shown that creating multiple client instances in a single JVM causes the client creation/usage to block after a certain number of created client instances. On my machine it was after 11 instances (= logical CPU cores - 1)

Analysing the problem showed that the creation of the message loop handler in the GrpcClient uses a default Executor for CompletableFutures. The default Executor is the ForkJoinPool.commonPool() (see https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html#commonPool-- )

This Executor should not be used for infinite loops or even blocking behavior like the message handler loop inside the GrpcClient. See: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinTask.html

ForkJoinTasks should perform relatively small amounts of computation. Large tasks should be split into smaller subtasks, usually via recursive decomposition. As a very rough rule of thumb, a task should perform more than 100 and less than 10000 basic computational steps, and should avoid indefinite looping. If tasks are too big, then parallelism cannot improve throughput. If too small, then memory and internal task maintenance overhead may overwhelm processing.

When creating multiple clients the common ForkJoinPool is finally exhausted (all waiting on the message queue) and additional message loops are not executed, until other message loops have been terminated.

Even if the GprcClient is usually a singleton instance in the JVM this behavior is unexpected after hard to discover. Especially if it occurs during (parallel) testing.

[YoEight]

Nice catch!