'Read timed out' exception when sending requests to /unixcommands endpoint of RSE API

Describe the bug

The following inttests have been failing since Monday 5th August. Whenever requests are made to the RSE API, to the /v1/unixcommands endpoint specifically, we are getting back a Read timed out exception. Requests to other endpoints like /v1/datasets work okay.

ZosFileDatasetLocalJava11UbuntuIsolatedRse > runs the ZosManagerFileDatasetIVT and fails in method datasetAttributeCheck
ZosFileDatasetLocalJava11UbuntuMvpRse > runs the ZosManagerFileDatasetIVT and fails in method datasetAttributeCheck
ZosFileDatasetLocalJava11UbuntuRse > runs the ZosManagerFileDatasetIVT and fails in method datasetAttributeCheck
ZosVSAMLocalJava11UbuntuIsolatedRse > runs the ZosManagerFileVSAMIVT and fails in method basicKSDSDefine
ZosVSAMLocalJava11UbuntuRse > runs the ZosManagerFileVSAMIVT and fails in method basicKSDSDefine
ZosVSAMLocalJava11UbuntuMvpRse > runs the ZosManagerFileVSAMIVT and fails in method basicKSDSDefine

Within these test methods, the RSE API is being sent a request to create or read details about datasets. The exception being thrown which causes the failure is below. These same tests successfully make requests to the RSE API server on the /datasets endpoint but any request to `/unixcommands/ fails with this exception.

12/08/2024 12:43:54.946 DEBUG main dev.galasa.zosrseapi.internal.RseapiRestApiProcessor.getCurrentRseapiServer - Using RSE API server on XXXX https://XXXXX
12/08/2024 12:43:54.946 TRACE main dev.galasa.zosrseapi.internal.RseapiImpl.addCommonHeaders - Adding HTTP header: accept: */*
12/08/2024 12:43:54.946 TRACE main dev.galasa.zosrseapi.internal.RseapiImpl.postJson - Request: POST https://XXXXX/rseapi/api/v1/unixcommands
12/08/2024 12:43:54.947 TRACE main dev.galasa.zosrseapi.internal.RseapiImpl.postJson - body: 
{"invocation":"tsocmd \"XXXXX","path":"/usr/bin"}
12/08/2024 12:46:55.010 ERROR main dev.galasa.zosrseapi.internal.RseapiImpl.postJson - dev.galasa.http.HttpClientException: Error executing http request
12/08/2024 12:46:55.012 ERROR main dev.galasa.zosrseapi.internal.RseapiRestApiProcessor.sendRequest - Problem with RSE API request
dev.galasa.zosrseapi.RseapiException: Problem with POST to RSE API server
    at dev.galasa.zosrseapi.internal.RseapiImpl.postJson(RseapiImpl.java:215) ~[-1723466580197:?]
    at dev.galasa.zosrseapi.internal.RseapiRestApiProcessor.sendRequest(RseapiRestApiProcessor.java:91) [-1723466580197:?]
    at dev.galasa.zosfile.rseapi.manager.internal.RseapiZosUnixCommand.execute(RseapiZosUnixCommand.java:43) [-1723466580132:?]
    at dev.galasa.zosfile.rseapi.manager.internal.RseapiZosDatasetAttributesListdsi.execListdsi(RseapiZosDatasetAttributesListdsi.java:323) [-1723466580132:?]
    at dev.galasa.zosfile.rseapi.manager.internal.RseapiZosDatasetAttributesListdsi.get(RseapiZosDatasetAttributesListdsi.java:124) [-1723466580132:?]
    at dev.galasa.zosfile.rseapi.manager.internal.RseapiZosDatasetImpl.retrieveAttibutes(RseapiZosDatasetImpl.java:716) [-1723466580132:?]
    at dev.galasa.zos.manager.ivt.ZosManagerFileDatasetIVT.datasetAttributeCheck(ZosManagerFileDatasetIVT.java:169) [-1723466578507:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at dev.galasa.framework.GenericMethodWrapper.invoke(GenericMethodWrapper.java:88) [-1723466576250:?]
    at dev.galasa.framework.TestMethodWrapper.invoke(TestMethodWrapper.java:61) [-1723466576250:?]
    at dev.galasa.framework.TestClassWrapper.runTestMethods(TestClassWrapper.java:216) [-1723466576250:?]
    at dev.galasa.framework.TestRunner.runTestClassWrapper(TestRunner.java:610) [-1723466576250:?]
    at dev.galasa.framework.TestRunner.runEnvironment(TestRunner.java:584) [-1723466576250:?]
    at dev.galasa.framework.TestRunner.createEnvironment(TestRunner.java:549) [-1723466576250:?]
    at dev.galasa.framework.TestRunner.generateEnvironment(TestRunner.java:506) [-1723466576250:?]
    at dev.galasa.framework.TestRunner.runTest(TestRunner.java:427) [-1723466576250:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at dev.galasa.boot.felix.FelixFramework.runTest(FelixFramework.java:248) [boot.jar:?]
    at dev.galasa.boot.Launcher.launch(Launcher.java:172) [boot.jar:?]
    at dev.galasa.boot.Launcher.main(Launcher.java:122) [boot.jar:?]
Caused by: dev.galasa.http.HttpClientException: Error executing http request
    at dev.galasa.http.internal.HttpClientImpl.execute(HttpClientImpl.java:864) ~[?:?]
    at dev.galasa.http.internal.HttpClientImpl.executeJsonRequest(HttpClientImpl.java:214) ~[?:?]
    at dev.galasa.http.internal.HttpClientImpl.postJson(HttpClientImpl.java:180) ~[?:?]
    at dev.galasa.zosrseapi.internal.RseapiImpl.postJson(RseapiImpl.java:208) ~[-1723466580197:?]
    ... 25 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
    at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
    at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
    at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) ~[?:?]
    at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[?:?]
    at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1364) ~[?:?]
    at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973) ~[?:?]
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[?:?]
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[?:?]
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[?:?]
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[?:?]
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[?:?]
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[?:?]
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[?:?]
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[?:?]
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[?:?]
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[?:?]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
    at dev.galasa.http.internal.HttpClientImpl.execute(HttpClientImpl.java:862) ~[?:?]
    at dev.galasa.http.internal.HttpClientImpl.executeJsonRequest(HttpClientImpl.java:214) ~[?:?]
    at dev.galasa.http.internal.HttpClientImpl.postJson(HttpClientImpl.java:180) ~[?:?]
    at dev.galasa.zosrseapi.internal.RseapiImpl.postJson(RseapiImpl.java:208) ~[-1723466580197:?]
    ... 25 more

Steps to reproduce

Generate a portfolio with galasactl runs prepare --bootstrap ${BOOTSTRAP} --portfolio test.yaml --stream inttests --package local --log -
Comment out all tests other than the 6 above.
Submit the test portfolio with galasactl runs submit --bootstrap ${BOOTSTRAP} --portfolio test.yaml --overridefile - --poll 5 --log -

Expected behavior

Requests to /unixcommands should work successfully and get a normal response. The test should be able to finish and Pass.

Additional context

The weekend prior to the tests beginning to fail, there were some updates on Plex2 to do with TLS and ports but we haven't been able to determine if this is related or a coincidence.

Things tried/investigated so far

The tests seem to only use the RSE API server on the MV2D image despite the CPS properties in the rseapi namespace pointing to the MV2C image.
I made some adjustments to several properties in the zos and rseapi namespaces and managed to test to use the RSE API server on MV2C instead, but the exception still occurred.
I tried to use the MV26 image next (trying all images we have access to on Plex2 to see if any difference), but for some reason I wasn't able to override the test to use MV26's RSE API server.

galasa-dev / projectmanagement