This issue is related to https://github.com/OpenLiberty/open-liberty/issues/27665
Under the covers, OLO sets up health probes (liveness/readiness) for the Semeru Cloud Compiler pod. These probes target the port used by Semeru Cloud Compiler for compilation requests (default 38400). This works fine as long as this port does not use SSL encryption.
However, OLO sets up traffic encryption between Liberty clients and Semeru Cloud Compiler servers. Starting with OpenSSL 3.0, the OpenSSL library will log errors if the peer (in this case the kubelet) does not inform the other party about the connection being closed.
What did you expect to see?
No errors in the logs of Semeru Cloud Compiler pod
What did you see instead?
The log for the Semeru Cloud Compiler pod shows errors like:
A0F04FE4FE7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:320:
Environment
Note: currently these errors are seen only with Java21 because Semeru containers with Java 21 include OpenSSL 3.0.
In contrast, Semeru containers for Java 8,11,17 include OpenSSL 1.1.1. This may change in the future.
Possible solution
Since currently there is no way to specify encryption for a TCP probe, the only solution is to send these probes to another Semeru Cloud Compiler which does not use encryption. Such a dedicated health port (default 38600) will be added to the Semeru Cloud Compiler in PR https://github.com/eclipse-openj9/openj9/pull/19107
It may be helpful to allow the user to further configure the health probes (such as the targeted port or the frequency).
Bug Report
What did you do?
This issue is related to https://github.com/OpenLiberty/open-liberty/issues/27665 Under the covers, OLO sets up health probes (liveness/readiness) for the Semeru Cloud Compiler pod. These probes target the port used by Semeru Cloud Compiler for compilation requests (default 38400). This works fine as long as this port does not use SSL encryption. However, OLO sets up traffic encryption between Liberty clients and Semeru Cloud Compiler servers. Starting with OpenSSL 3.0, the OpenSSL library will log errors if the peer (in this case the kubelet) does not inform the other party about the connection being closed.
What did you expect to see?
No errors in the logs of Semeru Cloud Compiler pod
What did you see instead?
The log for the Semeru Cloud Compiler pod shows errors like:
Environment
Note: currently these errors are seen only with Java21 because Semeru containers with Java 21 include OpenSSL 3.0. In contrast, Semeru containers for Java 8,11,17 include OpenSSL 1.1.1. This may change in the future.
Possible solution
Since currently there is no way to specify encryption for a TCP probe, the only solution is to send these probes to another Semeru Cloud Compiler which does not use encryption. Such a dedicated health port (default 38600) will be added to the Semeru Cloud Compiler in PR https://github.com/eclipse-openj9/openj9/pull/19107 It may be helpful to allow the user to further configure the health probes (such as the targeted port or the frequency).
Additional context
https://github.com/OpenLiberty/open-liberty/issues/27665