search

wso2 / docker-apim

Docker and Docker Compose resources for WSO2 API Manager
Apache License 2.0
143 stars 215 forks source link

SSL-related exceptions with connection to ssl://am-analytics:7712 #114

Open jeremy303 opened 6 years ago

jeremy303 commented 6 years ago

Description:

Receiving the following exceptions in api-manager, running the current master branch out-of-the-box:

api-manager_1   | [2018-03-28 21:00:54,702] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://am-analytics:7712
api-manager_1   | org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://am-analytics:7712
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:99)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
api-manager_1   |   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
api-manager_1   |   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
api-manager_1   |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
api-manager_1   |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
api-manager_1   |   at java.lang.Thread.run(Thread.java:748)
api-manager_1   | Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift exception
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:49)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:93)
api-manager_1   |   ... 6 more
api-manager_1   | Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
api-manager_1   |   at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
api-manager_1   |   at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
api-manager_1   |   at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:45)
api-manager_1   |   ... 7 more
api-manager_1   | Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
api-manager_1   |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
api-manager_1   |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
api-manager_1   |   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
api-manager_1   |   at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
api-manager_1   |   at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
api-manager_1   |   at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
api-manager_1   |   at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
api-manager_1   |   at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
api-manager_1   |   ... 11 more
api-manager_1   | Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
api-manager_1   |   at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
api-manager_1   |   at sun.security.validator.Validator.validate(Validator.java:260)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
api-manager_1   |   ... 21 more
api-manager_1   | Caused by: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
api-manager_1   |   at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
api-manager_1   |   at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
api-manager_1   |   ... 27 more
api-manager_1   | Caused by: java.security.SignatureException: Signature length not correct: got 256 but was expecting 128
api-manager_1   |   at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:189)
api-manager_1   |   at java.security.Signature$Delegate.engineVerify(Signature.java:1223)
api-manager_1   |   at java.security.Signature.verify(Signature.java:656)
api-manager_1   |   at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:444)
api-manager_1   |   at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
api-manager_1   |   at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
api-manager_1   |   at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
api-manager_1   |   ... 32 more
api-manager_1   | [2018-03-28 21:00:54,907]  WARN - DataEndpointGroup No receiver is reachable at reconnection, will try to reconnect every 30 sec
api-manager_1   | [2018-03-28 21:00:54,912] ERROR - DataEndpointConnectionWorker Thrift exception
api-manager_1   | org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift exception
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:49)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:93)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
api-manager_1   |   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
api-manager_1   |   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
api-manager_1   |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
api-manager_1   |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
api-manager_1   |   at java.lang.Thread.run(Thread.java:748)
api-manager_1   | Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
api-manager_1   |   at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
api-manager_1   |   at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
api-manager_1   |   at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
api-manager_1   |   at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:45)
api-manager_1   |   ... 7 more
api-manager_1   | Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1551)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1563)
api-manager_1   |   at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
api-manager_1   |   at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
api-manager_1   |   at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
api-manager_1   |   at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
api-manager_1   |   ... 11 more
api-manager_1   | Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
api-manager_1   |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
api-manager_1   |   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
api-manager_1   |   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
api-manager_1   |   at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
api-manager_1   |   at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
api-manager_1   |   at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
api-manager_1   |   ... 14 more
api-manager_1   | Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
api-manager_1   |   at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
api-manager_1   |   at sun.security.validator.Validator.validate(Validator.java:260)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
api-manager_1   |   at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
api-manager_1   |   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)
api-manager_1   |   ... 21 more
api-manager_1   | Caused by: java.security.cert.CertPathValidatorException: signature check failed
api-manager_1   |   at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
api-manager_1   |   at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
api-manager_1   |   at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
api-manager_1   |   at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
api-manager_1   |   ... 27 more
api-manager_1   | Caused by: java.security.SignatureException: Signature length not correct: got 256 but was expecting 128
api-manager_1   |   at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:189)
api-manager_1   |   at java.security.Signature$Delegate.engineVerify(Signature.java:1223)
api-manager_1   |   at java.security.Signature.verify(Signature.java:656)
api-manager_1   |   at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:444)
api-manager_1   |   at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
api-manager_1   |   at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
api-manager_1   |   at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
api-manager_1   |   ... 32 more
DilanUA commented 6 years ago

@HolySamosa Did you do any changes to the original deployment or is this an Out-of-the-box run ? Please do let us know steps to reproduce since we are not experiencing this issue at our side.

jeremy303 commented 6 years ago

Thanks, @DilanUA.

This is APIM-ISasKM-with-Analytics out-of-the-box-- almost. I did build the docker images locally using jdk1.8.0_161 and mysql-connector-java-5.1.46-bin.jar and modified the docker-compose.yml to pull the local images. Otherwise, no changes.

chamilad commented 6 years ago

@HolySamosa Could you try with a older JDK version, older than _151?

ichwill100 commented 6 years ago

@HolySamosa and @chamilad Is there any solution for this issue?

SureshG02 commented 5 years ago

Is this issue still Open ? I am also getting same error in my WSO2 APIM server for analtyics.

tekatool commented 5 years ago

am getting the same error... my setup apim-m 2.5.0 dockerized api-m admin + gateway docker on one machine and apim-analytics server on another.

** using OpenJDK 8 with AllowAll for hostname verification

enabled ssl debug and seeing the following in the api-m (client) logs...

trigger seeding of SecureRandom done seeding SecureRandom [2019-01-04 23:30:08,634] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://10.204.131.28:7714 org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://10.204.131.28:7714 at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:136) at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://10.204.131.28:7714 at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:81) at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:126) ... 6 more Caused by: org.apache.thrift.transport.TTransportException: Could not connect to 10.204.131.28 on port 7714 at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:237) at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:169) at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:64) ... 9 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:432) at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88) at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:233) ... 11 more

I can ping to the ip 10.204.131.28 from the client api-m.

Any prompt help is really appreciated...

Thanks

vmonsanto commented 5 years ago

the same issue for me, any solution.

ximeraz commented 4 years ago

@HolySamosa Did you import the certificate of Api Manager? I think is necessary to do that for the communication between this solutions. So remember that the user to authenticate api manager with analytics is admin and the password too.