Read timeout by micronaut client on configured enpoint

[ebrard]

I have configured the auth authentification along side with the call to a rest endpoint for permissions. Regularly the micronaut http clients runs into a time out as such:

2021-11-19 13:13:51,320 ERROR pGroup-1-7 .m.r.i.RecoveryInterceptor Type [org.akhq.utils.RestApiClaimProvider$Intercepted] executed with error: Read Timeout
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
    at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26)
    at io.micronaut.http.client.netty.DefaultHttpClient.lambda$null$36(DefaultHttpClient.java:1178)
    at io.reactivex.internal.operators.flowable.FlowableOnErrorNext$OnErrorNextSubscriber.onError(FlowableOnErrorNext.java:103)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onError(RxInstrumentedSubscriber.java:66)
    at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutSubscriber.onTimeout(FlowableTimeoutTimed.java:139)
    at io.reactivex.internal.operators.flowable.FlowableTimeoutTimed$TimeoutTask.run(FlowableTimeoutTimed.java:170)
    at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
    at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

which seems to mean that the rest client takes more than 10 sec to answer. On the rest server side we have Datadog APM configured and we see that all calls are being processed in a few ms, which makes me wonder if the issue could be in akhq. I have read somewhere that micronaut http client uses the main thread which can potentially conflict with akhq operations.

[ebrard]

I increased the micronaut client read time to 5 mins and still I see read timeout while hitting the rest service endpoint from another browser tab which seems to indicate that the rest server implemented can take extra calls.

[tchiotludo]

@ebrard can you retry with last version please ? Since there is a major upgrade from micronaut that don't use RxJava, it will be fine to test it again :pray:

[Monpoke]

Hey! I'm obtaining same issue, while trying to connect to OIDC provider.

I tried to debug a bit through JVM debug, and code has failed in Micronaut HttpClient... :-(

[ebrard]

So, we were still facing the issue. I believe that's because lots of operation are being done on the main thread(s) by akhq during a topic page view (topic listing, cg reverse lookup...). The following extra configuration seems to have fixed it once and for all (will confirm over time):

  netty:
    event-loops:
      other:
        num-threads: 10
        prefer-native-transport: true
  http:
    client:
      event-loop-group: other
      read-timeout: 1s