search

wso2 / kubernetes-apim

Kubernetes and Helm resources for WSO2 API Manager
Apache License 2.0
113 stars 216 forks source link

Errors connecting to analytics worker using custom CA #477

Open scholtz opened 3 years ago

scholtz commented 3 years ago

Description: I am testing setting up WSO2 Api Manager with Analytics worker with company certification authority in kubernetes environment.

I am receiving several error messages: UnknownHostException, unable to find valid certification path to requested target, Unsupported protocol: tcp. Currently only ssl supported

The most preferable solution is to use dns name such as wso2am-analytics-worker-service.wso2-api.svc.mydomain.com where wso2am-analytics-worker-service is the name of the service wso2-api is namespace, svc the service namespace, mydomain.com my custom domain.

I am able from the apim pod telnet the analytics worker with telnet wso2am-analytics-worker-service.wso2-api.svc.mydomain.com 7711

Also nslookup wso2am-analytics-worker-service.wso2-api.svc.mydomain.com shows the ip address of the service.

I am using coredns dns with rewrite rule rewrite name substring svc.mydomain.com svc.cluster.local. The service ip addresses in kubernetes are virtual ip addresses.

Therefore i believe there is a bug in java or wso that does not resolve dns on connection properly.

[2020-12-06 18:54:12,548] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7711
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7711
    at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:147) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) [org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7711
    at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:81) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) ~[commons-pool_1.5.6.wso2v1.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:137) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    ... 6 more
Caused by: org.apache.thrift.transport.TTransportException: Could not connect to wso2am-analytics-worker-service.wso2-api.svc.mydomain.com on port 7711
    at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:273) ~[libthrift_0.12.0.wso2v1.jar:?]
    at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:173) ~[libthrift_0.12.0.wso2v1.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:64) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) ~[commons-pool_1.5.6.wso2v1.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:137) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    ... 6 more
Caused by: java.net.UnknownHostException: wso2am-analytics-worker-service.wso2-api.svc.mydomain.com
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) ~[?:?]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
    at java.net.Socket.connect(Socket.java:609) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:289) ~[?:?]
    at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:148) ~[?:?]
    at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88) ~[?:?]
    at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:269) ~[libthrift_0.12.0.wso2v1.jar:?]
    at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:173) ~[libthrift_0.12.0.wso2v1.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:64) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212) ~[commons-pool_1.5.6.wso2v1.jar:?]
    at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:137) ~[org.wso2.carbon.databridge.agent_5.2.26.jar:?]
    ... 6 more

I have certificate for .mydomain.com from my company infra department. Therefore I must use the .svc.mydomain.com system for valid name resolution. When i try wso2am-analytics-worker-service.wso2-api.svc address which is also resolvable and connectable by telnet, it shows me error "unable to find valid certification path to requested target" which is reasonable because it is not in valid path. When i try to turn ssl off and connect to 7611 port, i get error "Unsupported protocol: tcp. Currently only ssl supported"

Suggested Labels: wso2-am, wso2-analytics-worker

Suggested Assignees:

Affected Product Version: I am using docker images: wso2am-analytics-worker:3.2.0 wso2am:3.2.0

OS, DB, other environment details and versions:
Kubernetes v1.19.4 wso2am 3.2.0

Steps to reproduce: apim configuration:

[apim.analytics]
enable = true
store_api_url = "http://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7443"
username = ".."
password = ".."

[[apim.analytics.url_group]]
analytics_url =["tcp://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7611"]
analytics_auth_url =["ssl://wso2am-analytics-worker-service.wso2-api.svc.mydomain.com:7711"]
type = "loadbalance"

worker configuration:


  dataReceivers:
  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Thrift
        # Data receiver properties
      properties:
        tcpPort: '7611'
        sslPort: '7711'

  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Binary
        # Data receiver properties
      properties:
        tcpPort: '9611'
        sslPort: '9711'
        tcpReceiverThreadPoolSize: '100'
        sslReceiverThreadPoolSize: '100'

  # Secure Vault Configuration
wso2.securevault:
  secretRepository:
    type: org.wso2.carbon.secvault.repository.DefaultSecretRepository
    parameters:
      privateKeyAlias: wso2carbon
      keystoreLocation: ${sys:carbon.home}/resources/security/securevault.jks
      secretPropertiesFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/secrets.properties
  masterKeyReader:
    type: org.wso2.carbon.secvault.reader.DefaultMasterKeyReader
    parameters:
      masterKeyReaderFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/master-keys.yaml

star.mydomain.com was replaced with wso2carbon alias certificate.

When using custom alias, even though it was present in all jks files in pod, the alias was not found. (JKS Alias in IS, APIM and other services works normally.. only in analytics worker there is some issue)

Related Issues:

HamzaOralK commented 3 years ago

Any luck on this issue?