search

wso2 / kubernetes-apim

Kubernetes and Helm resources for WSO2 API Manager
Apache License 2.0
114 stars 216 forks source link

wso2apim-with-analytics-apim deploy by helm can not start and always restart #158

Open superlee007 opened 5 years ago

superlee007 commented 5 years ago

Description: I have some troubles while deploy apim by helm in kubernetes.

Affected Product Version: 2.6.0

OS, DB, other environment details and versions:

Steps to reproduce:

  1. delete the file kubernetes-apim/helm/pattern-1/apim-with-analytics/templates/persistent-volumes.yaml because of we already have a nfs in kubernetes environment.
  2. edit the file kubernetes-apim/helm/pattern-1/apim-with-analytics/templates/wso2apim-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wso2apim-with-analytics-apim-deployment-volume-claim
  namespace : {{ .Values.namespace }}
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: "nfs-client-provisioner"
  # selector:
  #   matchLabels:
  #     purpose: apim-shared-deployment
  1. edit the file /kubernetes-apim/helm/pattern-1/mysql/values.yaml
persistence:
  enabled: true
  storageClass: "nfs-client-provisioner"
  accessMode: ReadWriteOnce
  size: 8Gi

however the server wso2apim-with-analytics-apim-deployment-8876c9c55-6xf4m can not start up and always restart.

the error info :

  1. warn

WARN - ApplicationManagementServiceComponent Templates directory not found at /home/wso2carbon/wso2am-2.6.0/repository/resources/identity/authntemplates

WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...

WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...

WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,org.wso2.carbon.mediation.initializer.services.SynapseEnvironmentService,

WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,

2.error 

ERROR - RegistryEventingServiceComponent Error Instantiating Registry Event Source

INFO - AndesConfigurationManager Main andes configuration located at : /home/wso2carbon/wso2am-2.6.0/repository/conf/broker.xml

INFO - RegistryEventingServiceComponent Successfully Initialized Eventing on Registry
FATAL - CarbonServerManager WSO2 Carbon initialization Failed
java.lang.ExceptionInInitializerError
    at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:520)
    at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:220)
    at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:105)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
    at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
    at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1230)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5370)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5668)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1700)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1690)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Shutdown in progress
    at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
    at java.lang.Runtime.addShutdownHook(Runtime.java:211)
    at org.wso2.carbon.core.multitenancy.MultitenantServerManager.<clinit>(MultitenantServerManager.java:43)
    ... 39 more
ERROR - CarbonCoreServiceComponent Failed clean up Carbon core
java.lang.NullPointerException
    at org.wso2.carbon.core.init.CarbonServerManager.stop(CarbonServerManager.java:947)
    at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.deactivate(CarbonCoreServiceComponent.java:118)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.deactivate(ServiceComponent.java:387)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.deactivate(ServiceComponentProp.java:161)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.dispose(ServiceComponentProp.java:387)
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.dispose(ServiceComponentProp.java:102)
    at org.eclipse.equinox.internal.ds.InstanceProcess.disposeInstances(InstanceProcess.java:344)
    at org.eclipse.equinox.internal.ds.InstanceProcess.disposeInstances(InstanceProcess.java:306)
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:368)
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:225)
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.unregisterServices(ServiceRegistry.java:635)
    at org.eclipse.osgi.framework.internal.core.BundleContextImpl.close(BundleContextImpl.java:88)
    at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:514)
    at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:566)
    at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1206)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:592)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
    at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
    at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)
    at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:692)
    at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:600)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.shutdown(EclipseStarter.java:400)
    at org.wso2.carbon.core.init.CarbonServerManager.shutdown(CarbonServerManager.java:851)
    at org.wso2.carbon.core.init.CarbonServerManager.shutdownGracefully(CarbonServerManager.java:885)
    at org.wso2.carbon.core.init.CarbonServerManager$4.run(CarbonServerManager.java:903)

...
ERROR - RegistryEventingServiceComponent Failed obtaining server configuration

INFO - CarbonServerManager Shutdown complete
INFO - CarbonServerManager Halting JVM

so, what is wrong with it, thanks!

superlee007 commented 5 years ago

I delete all the volume and re-deploy wso2-apim, and I found that wso2apim-with-analytics-apim always exit abnormally at the first start up, and when it restart the error like above saying appears.

I gress that may be it is because of the livenessProbe in kubernetes-apim/helm/pattern-1/apim-with-analytics/templates/wso2apim-deployment.yaml,so I comment out the code 

livenessProbe:
   exec:
     command:
     - /bin/sh
     - -c
     - nc -z localhost 9443
   initialDelaySeconds: 150
   periodSeconds: 10

and then delete all the pv and re-deploy, it works well.