search

wso2 / docker-is

Docker and Docker Compose resources for WSO2 Identity Server
Apache License 2.0
92 stars 156 forks source link

Error Joining Kubernetes 1.16 Clusters #201

Closed rgauss closed 4 years ago

rgauss commented 4 years ago

Description: When trying to join a Kubernetes 1.16 cluster the logs show the following errors:

[2020-05-22 14:48:59,883] [] ERROR {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Kubernetes membership initialization failed org.wso2.carbon.membership.scheme.kubernetes.exceptions.KubernetesMembershipSchemeException: Could not connect to Kubernetes API
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.connectAndRead(ApiBasedPodIpResolver.java:225)
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.resolveAddresses(ApiBasedPodIpResolver.java:135)
        at org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme.init(KubernetesMembershipScheme.java:94)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.initiateCustomMembershipScheme(HazelcastClusteringAgent.java:630)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.configureMembershipScheme(HazelcastClusteringAgent.java:611)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.createConfigForAxis2Mode(HazelcastClusteringAgent.java:403)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.loadHazelcastConfig(HazelcastClusteringAgent.java:449)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.init(HazelcastClusteringAgent.java:176)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.enableClustering(StartupFinalizerServiceComponent.java:299)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:187)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:323)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
        at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:99)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
        at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:529)
        at org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:305)
        at org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:973)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:254)
        at java.base/sun.security.ssl.SSLExtensions.<init>(SSLExtensions.java:90)
        at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestMessage.<init>(CertificateRequest.java:818)
        at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestConsumer.consume(CertificateRequest.java:922)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
        at org.wso2.carbon.membership.scheme.kubernetes.api.KubernetesApiEndpoint.read(KubernetesApiEndpoint.java:45)
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.connectAndRead(ApiBasedPodIpResolver.java:223)
        ... 48 more

[2020-05-22 14:48:59,901] [] ERROR {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - Cannot initialize cluster org.apache.axis2.clustering.ClusteringFault: Kubernetes membership initialization failed
        at org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme.init(KubernetesMembershipScheme.java:110)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.initiateCustomMembershipScheme(HazelcastClusteringAgent.java:630)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.configureMembershipScheme(HazelcastClusteringAgent.java:611)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.createConfigForAxis2Mode(HazelcastClusteringAgent.java:403)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.loadHazelcastConfig(HazelcastClusteringAgent.java:449)
        at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.init(HazelcastClusteringAgent.java:176)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.enableClustering(StartupFinalizerServiceComponent.java:299)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:187)
        at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:323)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
        at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:99)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
        at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:529)
        at org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:305)
        at org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:973)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: org.wso2.carbon.membership.scheme.kubernetes.exceptions.KubernetesMembershipSchemeException: Could not connect to Kubernetes API
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.connectAndRead(ApiBasedPodIpResolver.java:225)
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.resolveAddresses(ApiBasedPodIpResolver.java:135)
        at org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme.init(KubernetesMembershipScheme.java:94)
        ... 46 more
Caused by: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:254)
        at java.base/sun.security.ssl.SSLExtensions.<init>(SSLExtensions.java:90)
        at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestMessage.<init>(CertificateRequest.java:818)
        at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestConsumer.consume(CertificateRequest.java:922)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
        at org.wso2.carbon.membership.scheme.kubernetes.api.KubernetesApiEndpoint.read(KubernetesApiEndpoint.java:45)
        at org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver.connectAndRead(ApiBasedPodIpResolver.java:223)
        ... 48 more

which seems to be related to JDK-8236039 and has been fixed in a newer version of JDK.

Affected Product Version: 5.10.0

OS, DB, other environment details and versions:
OpenJDK 11.0.7

Steps to reproduce: Deploy to Kubernetes 1.16 and observe the logs

Related Issues: wso2/kubernetes-is#195

dhananjaya94 commented 4 years ago

Faced the same issue, Temporary workaround


      containers:
      - name: wso2is
        image: docker.wso2.com/wso2is:5.10.0
        args:
        - -Djdk.tls.client.protocols=TLSv1.2
chirangaalwis commented 4 years ago

Fixed in https://github.com/wso2/docker-is/pull/202. Appreciate your contribution @rgauss.

Closing the issue.