search

wso2 / product-is

Welcome to the WSO2 Identity Server source code! For info on working with the WSO2 Identity Server repository and contributing code, click the link below.
http://wso2.github.io/
Apache License 2.0
748 stars 728 forks source link

[Kubernetes] Failure to detect leaving IS Pod members during a forceful Pod deletion #12187

Closed chirangaalwis closed 1 week ago

chirangaalwis commented 3 years ago

Describe the issue: When deleting a random Identity Server Pod (along with its constituent containers) forcefully in a two node Identity Server Kubernetes deployment, the remaining Pod does not detect the leaving member, acknowledge its departure from the cluster or no cluster coordinator re-election.

The following sample logs cannot be detected in the remaining Pod.

[2021-07-28 07:54:59,446] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Elected this member [6a66cf34-b6d1-431d-adce-358d4f3cb53e] as the Coordinator node
[2021-07-28 07:54:59,511] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver} - Reading IP addresses from endpoints
[2021-07-28 07:54:59,512] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Member left: [UUID] c55d0877-8b56-4be2-8472-da4cb233b84f, [Address] /10.244.7.62:4000

Also, the newly spawned Identity Server Pod failed several times before successfully returning to Ready state. The Hazelcast cluster initialisation takes a considerable amount of time.

[2021-07-28 08:00:53,712] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster domain: wso2.carbon.domain
[2021-07-28 08:00:53,713] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Loading hazelcast configuration from axis2 clustering configuration
[2021-07-28 08:00:53,778] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Using kubernetes based membership management scheme
[2021-07-28 08:00:53,799] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Initializing kubernetes membership scheme...
[2021-07-28 08:00:53,801] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver} - Parameter KUBERNETES_API_SERVER not found, checking KUBERNETES_SERVICE_HOST & KUBERNETES_SERVICE_PORT_HTTPS
[2021-07-28 08:00:53,802] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver} - Kubernetes clustering configuration: [api-server] https://10.0.0.1:443 [namespace] wso2 [services] wso2is-pattern-1-identity-service [skip-master-ssl-verification] true
[2021-07-28 08:00:54,218] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver} - Reading IP addresses from endpoints
[2021-07-28 08:00:54,219] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Member added to cluster configuration: [container-ip] 10.244.1.34
[2021-07-28 08:00:54,220] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Kubernetes membership scheme initialized successfully
[2021-07-28 08:00:54,225] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast cluster is initializing...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/home/wso2carbon/wso2is-5.11.0/repository/components/plugins/hazelcast_3.12.2.wso2v1.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-07-28 08:03:02,296] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 128071ms
[2021-07-28 08:04:03,683] []  INFO {org.wso2.carbon.bpel.core.internal.BPELServiceComponent} - Shutting down BPEL Server........
[2021-07-28 08:04:03,684] []  INFO {org.apache.ode.bpel.engine.BpelServerImpl} - BPEL Server Stopped.
[2021-07-28 08:04:03,684] []  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered....
[2021-07-28 08:04:03,685] []  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down WSO2 Identity Server...
[2021-07-28 08:04:03,687] []  INFO {org.wso2.carbon.core.ServerManagement} - Starting to switch to maintenance mode...
[2021-07-28 08:04:03,691] []  INFO {org.wso2.carbon.bpel.core.ode.integration.BPELServerImpl} - BPEL Server shutdown completed.
[2021-07-28 08:04:03,691] []  INFO {org.wso2.carbon.core.ServerManagement} - Stopped all transport listeners
[2021-07-28 08:04:03,696] []  INFO {org.wso2.carbon.core.ServerManagement} - Waiting for request service completion...
[2021-07-28 08:04:03,703] []  INFO {org.wso2.carbon.core.ServerManagement} - All requests have been served.
[2021-07-28 08:04:03,703] []  INFO {org.wso2.carbon.core.ServerManagement} - Waiting for deployment completion...
[2021-07-28 08:04:03,726] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [78d34820-eb12-4d87-8d73-5c08358ced8d] - Host:10.244.7.64, Remote Host:null, Port: 4000, HTTP:9763, HTTPS:9443, Domain: wso2.carbon.domain, Sub-domain:worker, Active:true
[2021-07-28 08:04:03,731] [] ERROR {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - Cannot initialize cluster com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getOrCreateProxyFuture(ProxyRegistry.java:177)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getOrCreateProxy(ProxyRegistry.java:160)
    at com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:147)
    at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:384)
    at com.hazelcast.instance.HazelcastInstanceImpl.getTopic(HazelcastInstanceImpl.java:196)
    at com.hazelcast.instance.HazelcastInstanceProxy.getTopic(HazelcastInstanceProxy.java:109)
    at org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent.init(HazelcastClusteringAgent.java:238)
    at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.enableClustering(StartupFinalizerServiceComponent.java:299)
    at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.completeInitialization(StartupFinalizerServiceComponent.java:187)
    at org.wso2.carbon.core.internal.StartupFinalizerServiceComponent.serviceChanged(StartupFinalizerServiceComponent.java:323)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
    at org.wso2.carbon.server.admin.internal.ServerAdminServiceComponent.activate(ServerAdminServiceComponent.java:99)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
    at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:529)
    at org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:305)
    at org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118)
    at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:973)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)

[2021-07-28 08:04:03,797] []  INFO {org.wso2.carbon.healthcheck.api.core.internal.HealthMonitorServiceComponent} - Carbon health monitoring service is activated..
[2021-07-28 08:04:03,828] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/authenticationendpoint]
[2021-07-28 08:04:03,849] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/oauth2/uma/permission/v1.0]
[2021-07-28 08:04:03,861] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mex]
[2021-07-28 08:04:03,898] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/recovery/v0.9]
[2021-07-28 08:04:03,912] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/myaccount]
[2021-07-28 08:04:03,933] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/auth/v1.1]
[2021-07-28 08:04:03,952] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mexut]
[2021-07-28 08:04:03,960] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/smsotpauthenticationendpoint]
[2021-07-28 08:04:03,983] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/wso2]
[2021-07-28 08:04:03,999] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/totpauthenticationendpoint]
[2021-07-28 08:04:04,019] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/user/v1.0]
[2021-07-28 08:04:04,028] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/x509certificateauthenticationendpoint]
[2021-07-28 08:04:04,044] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/entitlement]
[2021-07-28 08:04:04,051] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/console]
[2021-07-28 08:04:04,067] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/users/v2/me/webauthn]
[2021-07-28 08:04:04,082] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/oauth2]
[2021-07-28 08:04:04,102] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/consent-mgt/v1.0]
[2021-07-28 08:04:04,109] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/STRATOS_ROOT]
[2021-07-28 08:04:04,130] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/scim2]
[2021-07-28 08:04:04,150] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/config-mgt/v1.0]
[2021-07-28 08:04:04,163] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/oauth2/uma/resourceregistration/v1.0]
[2021-07-28 08:04:04,177] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/health-check/v1.0]
[2021-07-28 08:04:04,183] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/lsp]
[2021-07-28 08:04:04,196] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/oauth2/dcr/v1.1]
[2021-07-28 08:04:04,206] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/template/mgt/v1.0.0]
[2021-07-28 08:04:04,220] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api/identity/oauth2/v1.0]
[2021-07-28 08:04:04,230] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/accountrecoveryendpoint]
[2021-07-28 08:04:04,241] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/emailotpauthenticationendpoint]
[2021-07-28 08:04:04,272] []  INFO {org.wso2.carbon.webapp.mgt.WebApplication} - Unloaded webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/api]
[2021-07-28 08:04:04,273] []  INFO {org.wso2.carbon.core.ServerManagement} - All deployment tasks have been completed.
[2021-07-28 08:04:04,274] []  INFO {org.wso2.carbon.core.ServerManagement} - Waiting for server task completion...
[2021-07-28 08:04:04,274] []  INFO {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} - Writing logs
[2021-07-28 08:04:04,301] []  INFO {org.wso2.carbon.humantask.core.HumanTaskServerShutdown} - Shutting down human task scheduler
[2021-07-28 08:04:04,302] []  INFO {org.wso2.carbon.core.ServerManagement} - All server tasks have been completed.
[2021-07-28 08:04:04,303] []  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutting down WSO2 Identity Server...
[2021-07-28 08:04:04,305] []  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutting down OSGi framework...
[2021-07-28 8:04:04,307]  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown complete
[2021-07-28 8:04:04,308]  INFO {org.wso2.carbon.core.init.CarbonServerManager} - Halting JVM

How to reproduce: We are using the official Identity Server pattern 1 deployment Kubernetes/Helm resources.

The Docker source used can be found from here.

The Kubernetes Pod probe configurations are as follows:

    Liveness:   exec [/bin/sh -c nc -z localhost 9443] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/sh -c nc -z localhost 9443] delay=60s timeout=1s period=15s #success=1 #failure=3
    Startup:    exec [/bin/sh -c nc -z localhost 9443] delay=60s timeout=1s period=5s #success=1 #failure=30

Plus, you can perform a forceful Pod deletion using the following command.

kubectl delete pod <pod-name> -n <namespace> --force

Expected behavior: The remaining Pods to detect the leaving member, acknowledge its departure from the cluster or cluster coordinator re-election.

[2021-07-28 07:54:59,446] []  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Elected this member [6a66cf34-b6d1-431d-adce-358d4f3cb53e] as the Coordinator node
[2021-07-28 07:54:59,511] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.resolver.ApiBasedPodIpResolver} - Reading IP addresses from endpoints
[2021-07-28 07:54:59,512] []  INFO {org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme} - Member left: [UUID] c55d0877-8b56-4be2-8472-da4cb233b84f, [Address] /10.244.7.62:4000

Environment information (Please complete the following information; remove any unnecessary fields) :

isharak commented 1 week ago

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!