[pattern1 - v3.2.0] DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2am-pattern-1-am-2-service:9711-

[gadeykarthik]

[2020-11-23 07:47:00,589] WARN - DataEndpointGroup No receiver is reachable at reconnection, will try to reconnect every 30 sec [2020-11-23 07:47:01,631] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2am-pattern-1-am-2-service:9711 org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://wso2am-pattern-1-am-2-service:9711 at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:147) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) [org.wso2.carbon.databridge.agent_5.2.26.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointException: Error while opening socket to wso2am-pattern-1-am-2-service:9711. Connection refused (Connection refused) at org.wso2.carbon.databridge.agent.endpoint.binary.BinarySecureClientPoolFactory.createClient(BinarySecureClientPoolFactory.java:75) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) ~[commons-pool_1.5.6.wso2v1.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:137) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] ... 6 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?] at java.net.Socket.connect(Socket.java:609) ~[?:?] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:289) ~[?:?] at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:148) ~[?:?] at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88) ~[?:?] at org.wso2.carbon.databridge.agent.endpoint.binary.BinarySecureClientPoolFactory.createClient(BinarySecureClientPoolFactory.java:58) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) ~[commons-pool_1.5.6.wso2v1.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:137) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?] ... 6 more [2020-11-23 07:47:30,589] WARN - DataEndpointGroup No receiver is reachable at reconnection, will try to reconnect every 30 sec [2020-11-23 07:56:55,648] WARN - AccessConfiguration Error loading properties from file: access-log.properties [2020-11-23 07:56:55,653] WARN - MiscellaneousUtil Error loading properties from a file at from the System defined location: access-log.properties [2020-11-23 07:56:55,656] WARN - MiscellaneousUtil Error loading properties from a file at from the System defined location: access-log.properties

[ashoka777]

same problem to me for api manager 3.2.0

[kalyan312]

same problem api manager 4.0.0

[ThijsFTW]

Same issues with API-Manager 4.1. Is this a configuration issue? Did anyone resolve it?

[pubudu538]

At the startup of the servers, you will notice this stack trace as each node is trying to connect with each other. SO this is normal at the startup. Are getting this after the startup? If that is the case, if one of the nodes not available, again you will see this stack trace.

[yasirjanjua]

@pubudu538 I have the same issue on api manager 4.0.0. I have distributed setup including two gateway nodes and single control-plane node. This error occurs every week and stays for at least 1 hour. please update the ticket as it's a critical issue

[TomasTokaMrazek]

I don't believe, that your instances work properly with this error present.

It's pretty obvious, what's happening. Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2am-pattern-1-am-2-service:9711. The APIM instance is trying to connect on SSL transport to untrusted hostname wso2am-pattern-1-am-2-service. Unless you have wso2cabron.jks certificate with Common Name set to the K8s service name, this connection can never be made as it's insecure by design. Hence why Connection refused.

You need to have trusted certificate generated for FQDN hostname, setup DNS to exposed load balancer for WSO2 (ingress does not support TCP/UDP transports in a simple way), and you have to replace all configurations with that hostname. In other words you need to have E2E TLS encryption between Kubernetes services. Good luck.

WSO2 products are not optimized to run in Kubernetes or Docker. The implementation here is half-assed and baked in, so they can market that their products are keeping up with modern practices. They don't.

[KarthikDev]

Hi @TomasTokaMrazek, We have generated the certificate with servicename FQDN and also short name in SAN's. However i can see the above exception but with unknown host Cannot borrow client for ssl://api-manager1.api-gateway-t.svc.cluster.local:9711 Caused by: java.net.UnknownHostException: api-manager1.api-gateway-t.svc.cluster.local

ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://api-manager2.api-gateway-t.svc.cluster.local:9711 Caused by: java.net.UnknownHostException: api-manager2.api-gateway-t.svc.cluster.local

WARN - DataEndpointGroup No receiver is reachable at URL Endpoint/Endpoints [tcp://api-manager1.api-gateway-t.svc.cluster.local:9611] Caused by: java.net.SocketTimeoutException: Read timed out

DataEndpointGroup No receiver is reachable at URL Endpoint/Endpoints [tcp://api-manager1.api-gateway-t.svc.cluster.local:9611] java.net.SocketTimeoutException: Read timed out

[TomasTokaMrazek]

@KarthikDev Do you have property set up DNS or entry in hosts table? Unknown host = cannot resolve the name to IP.

[hitechie17]

APIM 4.2.0 Windows 11 ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://10.254.87.185:9711 org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://10.254.87.185:9711 at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:147) ~[org.wso2.carbon.databridge.agent_5.3.5.jar:?] at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) ~[org.wso2.carbon.databridge.agent_5.3.5.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointException: Error while opening socket to 10.254.87.185:9711. Permission denied: connect