Closed anmol1vw13 closed 1 year ago
@anmol1vw13 As @pderop wrote in his comment https://github.com/reactor/reactor-netty/issues/2499#issuecomment-1260706911 please upgrade to the latest released versions for Reactor/Spring
I've enabled debug logs and here's my observation.
Things work fine, if there's continuous traffic. But once there's a complete halt i.e. the API calls using netty is not being made, I see the following log
USER_EVENT: SslCloseCompletionEvent(SUCCESS)
READ COMPLETE
All the subsequent requests fail thereafter.
When does this USER_EVENT, SslCloseCompletionEvent get fired and how do I auto recover the connection, when the next request comes through?
@violetagg Updating spring to the latest version is not an option at the moment. There could be a lot of breaking changes. It would be great if you could help out with the SslCloseCompletionEvent.. Couldn't find much on it online.
@anmol1vw13 That's unfortunate ... you are missing an important fix in 1.0.18
release https://github.com/reactor/reactor-netty/releases/tag/v1.0.18
@anmol1vw13 May be as a workaround for your case (as you cannot upgrade) you can configure maxIdleTime
for the connection pool - More information you can find here https://github.com/reactor/reactor-netty/issues/1318#issuecomment-702668918
@violetagg I found a patch version of spring-boot-starter-webflux and it makes use of reactor-netty-http version 1.0.19. Here's my dependency tree
[INFO] +- org.springframework.boot:spring-boot-starter-webflux:jar:2.5.14:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-json:jar:2.5.14:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.12.6:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.6:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.6:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:2.5.14:compile
[INFO] | | \- io.projectreactor.netty:reactor-netty-http:jar:1.0.19:compile
[INFO] | | +- io.netty:netty-codec-http2:jar:4.1.77.Final:compile
[INFO] | | +- io.netty:netty-resolver-dns:jar:4.1.77.Final:compile
[INFO] | | | \- io.netty:netty-codec-dns:jar:4.1.77.Final:compile
[INFO] | | +- io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.77.Final:compile
[INFO] | | | \- io.netty:netty-resolver-dns-classes-macos:jar:4.1.77.Final:compile
[INFO] | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.77.Final:compile
[INFO] | | | +- io.netty:netty-transport-native-unix-common:jar:4.1.77.Final:compile
[INFO] | | | \- io.netty:netty-transport-classes-epoll:jar:4.1.77.Final:compile
[INFO] | | \- io.projectreactor.netty:reactor-netty-core:jar:1.0.19:compile
[INFO] | | \- io.netty:netty-handler-proxy:jar:4.1.77.Final:compile
[INFO] | | \- io.netty:netty-codec-socks:jar:4.1.77.Final:compile
...
This doesn't seem to work though because after a few minutes of inactivity I still see the SslCloseCompletionEvent in the logs, after which all the subsequent request fail with SslClosedEngineException.
Can you help me understanding when and why does SslCloseCompletionEvent occur? I mean even if the SSLEngine gets closed, how can we recover when the next external API call happens?
Setting maxIdleTime got things working. This exercise also helped me gain a lot of understanding on how reactor-netty-http works. It's a beast and does a lot of things to ensure the requests are quick.
@violetagg Instead of using environment variables, I'm using a ConnectionProvider as shown below. Please let me know if this is the right way to set it or if I need to change anything.
@Bean
public WebClient apiClient() {
/*
* Setting maxIdleTime as 10s, because servers usually have a keepAliveTimeout
* of 60s, after which the connection gets closed.
* If the connection pool has any connection which has been idle for over 10s, it
* will be evicted from the pool.
* Refer https://github.com/reactor/reactor-netty/issues/1318#issuecomment-702668918
*/
ConnectionProvider connectionProvider = ConnectionProvider.builder("connectionProvider")
.maxIdleTime(Duration.ofSeconds(10))
.build();
HttpClient httpClient = HttpClient.create(connectionProvider)
.wiretap("reactor.netty.http.client.HttpClient", LogLevel.DEBUG, AdvancedByteBufFormat.TEXTUAL)
.responseTimeout(Duration.of(5, ChronoUnit.SECONDS));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
}
@anmol1vw13 Yes that's the correct way of configuring the connection pool. You may want to configure max connections because with the configuration above you will have only 16 connections. Please check the documentation for more information https://projectreactor.io/docs/netty/release/reference/index.html#_connection_pool_2
@violetagg How about I use this configuration, as mentioned in the document!
ConnectionProvider provider =
ConnectionProvider.builder("custom")
.maxConnections(50)
.maxIdleTime(Duration.ofSeconds(20))
.maxLifeTime(Duration.ofSeconds(60))
.pendingAcquireTimeout(Duration.ofSeconds(60))
.evictInBackground(Duration.ofSeconds(120))
.build();
Can you explain what pending acquire timeout means?
@anmol1vw13 This is just an example, you have to test the configuration with your solution and to see which settings are more appropriate for your use case.
pending acquire timeout
If there is no free connection in the pool, this is the time that we will wait for a connection to become free.
@anmol1vw13 ,
Hi,
We are about to merge the #2518 PR, which may avoid the "SSLEngine closed already" exception in your case. In the meantime, you can try to use the work around described by Violeta.
However, using the PR, if the server closes the connection while the client is currently writing a request to the connection, or is currently waiting for the response, then the client will get a PrematureCloseException or an AbortedException (this we can't avoid). thanks.
I'm facing the following SSLEngine issue with negligible load.
The connection observed an error io.netty.handler.ssl.SslClosedEngineException: SSLEngine closed already
Stack: spring-boot-starter-parent 2.5.6 spring-boot-starter-webflux 2.5.6
Here's the trace
The web client bean that is used is as shown below
A sample API call using webclient
Dependency tree