Got http calls ERR_TIMED_OUT (TIMEOUT) on Spring Boot 2.7.x when enable SSL (https)

[tinalin314159]

I have an API server which built with Spring Boot 2.4.x on public domain and public IP which serve APIs running several years normal.

After 2023-01-01, user reports front-end web fails to call API after 30 seconds of the first API calls. (The first API call is successfully)

We tried use other RESTful client software to call API and get same result. It's not the front-end web problem.

We trace the error with Chrome developer mode (DevTools) - Network tab, get (failed) net::ERR_TIMED_OUT in status field.

We do not change any our code, any SpringBoot version and settings or OS updates in server side in two months before 2023-01-01.

We guess it may cause by browser or OS updates of client side. But it cannot solve by retaining the old version of browser or OS.

We guess it may Tomcat version or SpringBoot version is not compatible with current browser or OS.

※ no proxy server between client and server

It's a huge modification difference when upgrading to Tomcat 10 and/or SpringBoot 3.x above.

So we try to upgrade to the SpringBoot 2.7.x but it remains occurs when enable SSL (https).

We tried the status is stable http calls without ERR_TIMED_OUT when SSL (https) is not set in any version of SpringBoot/Tomcat.

Did anyone face the same problem?

Environment: Spring Boot 2.4.2 (Tomcat 9.0.43) and 2.7.8 (Tomcat 9.0.71) which is the latest version of Spring Boot 2.7.x) have same problem

Settings in properties file: (SSL setup section in properties file, and the SSL (https) is successfully on) [It's also the only difference between stable status and ERR_TIMED_OUT status]

server.port=8081
server.ssl.enabled=true
server.ssl.key-store=classpath:keystore.p12
server.ssl.key-store-password={my_key_store_password}
server.ssl.key-store-type=PKCS12
server.ssl.key-alias={my_alias}

Before the logging DEBUG mode enabled, the SpringBoot console or log file do not have any log when ERR_TIMED_OUT occurs. It means that the http call connection do not send to SpringBoot API service, the connection eaten by Tomcat.

After we setup the logging DEBUG mode enabled, we got the log when ERR_TIMED_OUT occurs as below:

2023-02-15 05:41:53,658 DEBUG [https-jsse-nio-8081-exec-2] Http11Processor : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@523bac8a:org.apache.tomcat.util.net.SecureNioChannel@3326d48e:java.nio.channels.SocketChannel[connected local=/{my_server_ip}::8081 remote=/{my_client_ip}:55823]], Status in: [CONNECT_FAIL], State out: [CLOSED]
2023-02-15 05:41:53,658 DEBUG [https-jsse-nio-8081-exec-2] LimitLatch : Counting down[https-jsse-nio-8081-exec-2] latch=1
2023-02-15 05:41:53,658 DEBUG [https-jsse-nio-8081-exec-2] NioEndpoint : Calling [org.apache.tomcat.util.net.NioEndpoint@3e576e09].closeSocket([org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@523bac8a:org.apache.tomcat.util.net.SecureNioChannel@3326d48e:java.nio.channels.SocketChannel[connected local=/{my_server_ip}:8081 remote=/{my_client_ip}:55823]])

The weird part is this error situation is only happen when we add the ssl settings as I described above.

[wilkinsona]

Thanks for the report. Spring Boot isn't responsible for establishing and maintaining HTTP connections, with or without SSL. That's done by code in Tomcat, building on top of the JDK and underlying OS. I would recommend asking on the Tomcat users mailing list for some guidance on how to diagnose SSL problems with Tomcat. Someone there should be able to explain what those log messages mean and may have some suggestions to determining why they're happening.

[tinalin314159]

I prepare a test-apis project to demo this problem, it only use the sample code from [spring.io] https://spring.io/guides/gs/rest-service/ without any other settings or dependencies to re-produce this problem. All you need to do is generate a PKCS12 type of keystore.p12 file of some domain name put at /test-apis/src/main/resources folder to test it The sample code is put on GitHub https://github.com/tinalin314159/test-apis I think this is not simply the Tomcat problem and I cannot re-produce it without SpringBoot package with just only Tomcat.

The way I generate the PKCS12 type of keystore.p12 : openssl pkcs12 -export -in fullchain.pem -inkey privatekey.pem -out keystore.p12 -name tomcat -CAfile chain.pem -caname root

[wilkinsona]

I can't reproduce the problem with the supplied sample, either running it locally or in a container built using the Dockerfile. In both cases I was able to make over 100 requests to the app using Chrome and none timed out.

As I suggested above, please ask the Tomcat community for some help as they will hopefully have the necessary expertise to diagnose SSL-level problems with Tomcat's endpoints.

[tinalin314159]

Thanks for your kindly help and trial. I'd already rule out the problem is not related to SpringBoot any version.

I focus on the problem maybe is the PKCS12 type of keystore.p12 file may cause the Tomcat thread failed problem (TIMEOUT) due to the Chrome or OS updates after 2023/01/01.

Suddenly, I found a phase on https://whatsmychaincert.com/ mention about

You do not need to include the root certificate in the certificate chain that you serve, since clients already have the root certificate in their trust stores. Including the root is inefficient since it increases the size of the SSL handshake.

After the trial of remove root cert. during the generation of PKCS12 type of keystore.p12 file it works. The SpringBoot API do not have any timeout anymore.

I guess there may have changes on Chrome or OS during the SSL handshake after 2023/01/01. If there have overhead during SSL handshake. The thread of Tomcat will dead after the first time successful API calls. Because we do not change SpringBoot version, Tomcat version and cert. before 2023/01/01. The only changes is Chrome or OS updates.

Thank you very much for your help to let me clarify the problem where is and solved.