search

wso2 / kubernetes-apim

Kubernetes and Helm resources for WSO2 API Manager
Apache License 2.0
114 stars 216 forks source link

Helm pattern 1 - wso2apim-with-analytics-apim-deployment CrashLoopBackOff #252

Open brjlpz opened 5 years ago

brjlpz commented 5 years ago

Description: Deploying WSO2 APIM with Helm pattern-1 in a AKS cluster, the pod wso2apim-with-analytics-apim-deployment stuck in a loop: 

kubectl get pods -n wso2                                                                                                                                              
NAME                                                              READY   STATUS    RESTARTS   AGE
wso2apim-with-analytics-apim-analytics-deployment-7fdb6d98wfwzk   1/1     Running   0          19h
wso2apim-with-analytics-apim-deployment-fcd674f6b-479hx           0/1     Running   2          8m51s
wso2apim-with-analytics-rdbms-service-mysql-8686c6475d-snx5d      1/1     Running   0          24h
kubectl logs -f wso2apim-with-analytics-apim-deployment-fcd674f6b-479hx -n wso2                                                                                       
JAVA_HOME environment variable is set to /opt/java/openjdk
CARBON_HOME environment variable is set to /home/wso2carbon/wso2am-2.6.0
Using Java memory options: -Xms256m -Xmx1024m
[2019-09-12 06:49:15,715]  INFO - QpidBundleActivator Setting BundleContext in PluginManager
[2019-09-12 06:49:16,679]  INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2019-09-12 06:49:16,679]  INFO - CarbonCoreActivator Operating System : Linux 4.15.0-1052-azure, amd64
[2019-09-12 06:49:16,680]  INFO - CarbonCoreActivator Java Home        : /opt/java/openjdk/jre
[2019-09-12 06:49:16,680]  INFO - CarbonCoreActivator Java Version     : 1.8.0_212
[2019-09-12 06:49:16,680]  INFO - CarbonCoreActivator Java VM          : OpenJDK 64-Bit Server VM 25.212-b03,AdoptOpenJDK
[2019-09-12 06:49:16,680]  INFO - CarbonCoreActivator Carbon Home      : /home/wso2carbon/wso2am-2.6.0
[2019-09-12 06:49:16,680]  INFO - CarbonCoreActivator Java Temp Dir    : /home/wso2carbon/wso2am-2.6.0/tmp
[2019-09-12 06:49:16,681]  INFO - CarbonCoreActivator User             : wso2carbon, en-US, GMT
[2019-09-12 06:49:16,788]  INFO - DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' won't be registered as an internal crypto provider. Please set the secret if the provider needs to be registered.
[2019-09-12 06:49:17,046]  INFO - KafkaEventAdapterServiceDS Successfully deployed the Kafka output event adaptor service
[2019-09-12 06:49:17,168]  INFO - TemplateDeployerServiceTrackerDS Successfully deployed the execution manager tracker service
[2019-09-12 06:49:19,443]  INFO - ConsentManagerComponent ConsentManagerComponent is activated.
[2019-09-12 06:49:19,678]  INFO - BinaryDataReceiver Started Binary SSL Transport on port : 9711
[2019-09-12 06:49:19,688]  INFO - BinaryDataReceiver Started Binary TCP Transport on port : 9611
[2019-09-12 06:49:19,692]  INFO - DataBridgeDS Successfully deployed Agent Server 
[2019-09-12 06:49:20,630]  INFO - EmbeddedRegistryService Configured Registry in 276ms
[2019-09-12 06:49:20,701]  INFO - EmbeddedRegistryService Connected to mount at govregistry in 20ms
[2019-09-12 06:49:20,790]  INFO - EmbeddedRegistryService Connected to mount at govregistry in 1ms
[2019-09-12 06:49:20,895]  INFO - RegistryCoreServiceComponent Registry Mode    : READ-WRITE
[2019-09-12 06:49:20,914]  INFO - AndesConfigurationManager Main andes configuration located at : /home/wso2carbon/wso2am-2.6.0/repository/conf/broker.xml
[2019-09-12 06:49:21,151]  INFO - AndesConfigurationManager Main andes configuration located at : /home/wso2carbon/wso2am-2.6.0/repository/conf/broker.xml
[2019-09-12 06:49:29,842]  INFO - TaglibUriRule TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined
[2019-09-12 06:50:25,571]  INFO - JmxReporterBuilder Creating JMX reporter for Metrics with domain 'org.wso2.carbon.metrics'
[2019-09-12 06:50:25,576]  INFO - AbstractReporter Started JMX reporter for Metrics
[2019-09-12 06:50:26,778]  INFO - SolrClient Default Embedded Solr Server Initialized
[2019-09-12 06:50:27,073]  INFO - UserStoreMgtDSComponent Carbon UserStoreMgtDSComponent activated successfully.
[2019-09-12 06:50:28,437]  INFO - ClusterBuilder Clustering has been disabled
[2019-09-12 06:50:28,832]  INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2019-09-12 06:50:28,833]  INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2019-09-12 06:50:29,526]  INFO - WebsocketTransportSender WSS Sender started
[2019-09-12 06:50:29,533]  INFO - PassThroughHttpSender Initializing Pass-through HTTP/S Sender...
[2019-09-12 06:50:29,580]  INFO - PassThroughHttpSender Pass-through HTTP Sender started...
[2019-09-12 06:50:29,584]  INFO - PassThroughHttpSSLSender Initializing Pass-through HTTP/S Sender...
[2019-09-12 06:50:29,613]  INFO - ClientConnFactoryBuilder customSSLProfiles configuration is loaded from path: /home/wso2carbon/wso2am-2.6.0/repository/resources/security/sslprofiles.xml
[2019-09-12 06:50:29,617]  INFO - ClientConnFactoryBuilder HTTPS Loading custom SSL profiles for the HTTPS sender
[2019-09-12 06:50:29,645]  INFO - ClientConnFactoryBuilder HTTPS Custom SSL profiles initialized for 1 servers
[2019-09-12 06:50:29,656]  INFO - PassThroughHttpSSLSender Pass-through HTTPS Sender started...
[2019-09-12 06:50:29,674]  INFO - WebsocketTransportSender WS Sender started
[2019-09-12 06:50:29,682]  INFO - PassThroughHttpListener Initializing Pass-through HTTP/S Listener...
[2019-09-12 06:50:29,699]  INFO - PassThroughHttpSSLListener Initializing Pass-through HTTP/S Listener...
[2019-09-12 06:50:29,814]  INFO - ModuleDeployer Deploying module: addressing-1.6.1-wso2v28 - file:/home/wso2carbon/wso2am-2.6.0/repository/deployment/client/modules/addressing-1.6.1-wso2v28.mar
[2019-09-12 06:50:30,212]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.andes.admin-3.2.44 - 
[2019-09-12 06:50:30,241]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.andes.admin.mqtt-3.2.44 - 
[2019-09-12 06:50:30,267]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.andes.event.admin-3.2.44 - 
[2019-09-12 06:50:31,104]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.application.authentication.framework-5.12.153 - 
[2019-09-12 06:50:31,196]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.discovery-6.0.53 - 
[2019-09-12 06:50:31,247]  INFO - DeploymentInterceptor Deploying Axis2 service: AccountCredentialMgtConfigService {super-tenant}
[2019-09-12 06:50:31,252]  INFO - DeploymentInterceptor Deploying Axis2 service: UserIdentityManagementAdminService {super-tenant}
[2019-09-12 06:50:31,258]  INFO - DeploymentInterceptor Deploying Axis2 service: UserInformationRecoveryService {super-tenant}
[2019-09-12 06:50:31,262]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.mgt-5.12.153 - 
[2019-09-12 06:50:31,355]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.webfinger-6.0.53 - 
[2019-09-12 06:50:31,475]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.processor-4.6.105 - 
[2019-09-12 06:50:31,482]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.store-4.6.105 - 
[2019-09-12 06:50:31,769]  INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.registry.ws.api-4.6.38 - 
[2019-09-12 06:50:32,874]  INFO - CarbonServerManager Repository       : /home/wso2carbon/wso2am-2.6.0/repository/deployment/server/
[2019-09-12 06:50:32,887]  INFO - KafkaEventAdapterServiceHolder Kafka input event adaptor waiting for dependent configurations to load
[2019-09-12 06:50:32,973]  INFO - QueueManagerServiceDS Successfully created the queue manager service
[2019-09-12 06:50:32,980]  INFO - SubscriptionManagerServiceDS Successfully created the subscription manager service
[2019-09-12 06:50:33,218]  INFO - ModuleDeployer Deploying module: addressing-1.6.1-wso2v28 - file:/home/wso2carbon/wso2am-2.6.0/repository/deployment/client/modules/addressing-1.6.1-wso2v28.mar
[2019-09-12 06:50:33,378]  INFO - ModuleDeployer Deploying module: addressing-1.6.1-wso2v28 - file:/home/wso2carbon/wso2am-2.6.0/repository/deployment/client/modules/addressing-1.6.1-wso2v28.mar
[2019-09-12 06:50:33,625]  INFO - TenantLoadingConfig Using tenant lazy loading policy...
[2019-09-12 06:50:33,655]  INFO - PermissionUpdater Permission cache updated for tenant -1234
[2019-09-12 06:50:34,050]  WARN - ApplicationManagementServiceComponent Templates directory not found at /home/wso2carbon/wso2am-2.6.0/repository/resources/identity/authntemplates
[2019-09-12 06:50:34,070]  INFO - DefaultKeyValidationHandler org.wso2.carbon.apimgt.keymgt.handlers.DefaultKeyValidationHandler Initialised
[2019-09-12 06:50:34,071]  INFO - APIKeyValidationService Initialised KeyValidationHandler instance successfully
[2019-09-12 06:50:34,085]  INFO - APIKeyMgtServiceComponent Started thrift key mgt service at port:10397
[2019-09-12 06:50:34,198]  INFO - UserStoreMgtDSComponent Claim manager set for class org.wso2.carbon.user.core.common.DefaultRealm
[2019-09-12 06:50:34,202]  INFO - UserStoreMgtDSComponent Claim manager set for class org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager
[2019-09-12 06:50:34,282]  INFO - SessionDataStore Thread pool size for temporary authentication context data delete task: 20
[2019-09-12 06:50:34,476]  INFO - OAuth2ServiceComponent PKCE Support enabled.
[2019-09-12 06:50:36,279]  INFO - ServiceBusInitializer Starting ESB...
[2019-09-12 06:50:36,297]  INFO - ServiceBusInitializer Initializing Apache Synapse...
[2019-09-12 06:50:36,308]  INFO - SynapseControllerFactory Using Synapse home : /home/wso2carbon/wso2am-2.6.0/.
[2019-09-12 06:50:36,390]  INFO - SynapseControllerFactory Using synapse.xml location : /home/wso2carbon/wso2am-2.6.0/././repository/deployment/server/synapse-configs/default
[2019-09-12 06:50:36,391]  INFO - SynapseControllerFactory Using server name : localhost
[2019-09-12 06:50:36,395]  INFO - SynapseControllerFactory The timeout handler will run every : 15s
[2019-09-12 06:50:36,401]  INFO - Axis2SynapseController Initializing Synapse at : Thu Sep 12 06:50:36 GMT 2019
[2019-09-12 06:50:36,408]  INFO - CarbonSynapseController Loading the mediation configuration from the file system
[2019-09-12 06:50:36,411]  INFO - MultiXMLConfigurationBuilder Building synapse configuration from the synapse artifact repository at : ././repository/deployment/server/synapse-configs/default
[2019-09-12 06:50:36,456]  INFO - XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
[2019-09-12 06:50:36,770]  INFO - DependencyTracker Sequence : dispatchSeq was added to the Synapse configuration successfully
[2019-09-12 06:50:36,857]  INFO - DependencyTracker Sequence : fault was added to the Synapse configuration successfully
[2019-09-12 06:50:36,885]  INFO - DependencyTracker Sequence : main was added to the Synapse configuration successfully
[2019-09-12 06:50:36,915]  INFO - DependencyTracker Sequence : outDispatchSeq was added to the Synapse configuration successfully
[2019-09-12 06:50:36,954]  INFO - DependencyTracker Sequence : _auth_failure_handler_ was added to the Synapse configuration successfully
[2019-09-12 06:50:36,977]  INFO - DependencyTracker Sequence : _build_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,012]  INFO - DependencyTracker Sequence : _cors_request_handler_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,035]  INFO - DependencyTracker Sequence : _production_key_error_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,061]  INFO - DependencyTracker Sequence : _resource_mismatch_handler_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,089]  INFO - DependencyTracker Sequence : _sandbox_key_error_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,118]  INFO - DependencyTracker Sequence : _threat_fault_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,149]  INFO - DependencyTracker Sequence : _throttle_out_handler_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,179]  INFO - DependencyTracker Sequence : _token_fault_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,290]  INFO - DependencyTracker Proxy service : WorkflowCallbackService was added to the Synapse configuration successfully
[2019-09-12 06:50:37,417]  INFO - DependencyTracker API : _WSO2AMAuthorizeAPI_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,466]  INFO - DependencyTracker API : _WSO2AMRevokeAPI_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,514]  INFO - DependencyTracker API : _WSO2AMTokenAPI_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,559]  INFO - DependencyTracker API : _WSO2AMUserInfoAPI_ was added to the Synapse configuration successfully
[2019-09-12 06:50:37,649]  INFO - DependencyTracker Inbound Endpoint : SecureWebSocketEP was added to the Synapse configuration successfully
[2019-09-12 06:50:37,701]  INFO - DependencyTracker Inbound Endpoint : WebSocketInboundEndpoint was added to the Synapse configuration successfully
[2019-09-12 06:50:37,702]  INFO - SynapseConfigurationBuilder Loaded Synapse configuration from the artifact repository at : ././repository/deployment/server/synapse-configs/default
[2019-09-12 06:50:37,704]  INFO - DependencyTracker Local entry : SERVER_HOST was added to the Synapse configuration successfully
[2019-09-12 06:50:37,704]  INFO - DependencyTracker Local entry : SERVER_IP was added to the Synapse configuration successfully
[2019-09-12 06:50:37,707]  INFO - Axis2SynapseController Loading mediator extensions...
[2019-09-12 06:50:49,686]  INFO - SchemaBuilder XACML policy schema loaded successfully.
[2019-09-12 06:51:12,890]  INFO - CarbonEventManagementService Starting polling event receivers
[2019-09-12 06:51:33,411]  WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:51:33,411]  WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:51:33,578]  WARN - AppDeployerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.synapse.service.SynapseAppDeployerService,org.wso2.carbon.mediation.initializer.services.SynapseEnvironmentService,
[2019-09-12 06:51:33,627]  WARN - StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.application.deployer.service.CappDeploymentService,org.wso2.carbon.server.admin.common.IServerAdmin,org.wso2.carbon.throttling.agent.ThrottlingAgent,
[2019-09-12 06:51:48,413]  WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:51:48,415]  WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:52:03,415]  WARN - BlockingConditionRetriever Failed retrieving Blocking Conditions from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:52:03,417]  WARN - KeyTemplateRetriever Failed retrieving throttling data from remote endpoint: Connection refused (Connection refused). Retrying after 15 seconds...
[2019-09-12 06:52:06,349]  INFO - CarbonServerManager Shutdown hook triggered....
[2019-09-12 06:52:06,350]  INFO - CarbonServerManager Gracefully shutting down WSO2 API Manager...
[2019-09-12 06:52:06,351]  INFO - ServerManagement Starting to switch to maintenance mode...
[2019-09-12 06:52:06,351]  INFO - ServerManagement Stopped all transport listeners
[2019-09-12 06:52:06,352]  INFO - ServerManagement Waiting for request service completion...
[2019-09-12 06:52:06,354]  INFO - ServerManagement All requests have been served.
[2019-09-12 06:52:06,354]  INFO - ServerManagement Waiting for deployment completion...
[2019-09-12 06:52:06,360]  INFO - ServerManagement All deployment tasks have been completed.
[2019-09-12 06:52:06,360]  INFO - ServerManagement Waiting for server task completion...
[2019-09-12 06:52:06,364]  INFO - RegistryCoreServiceComponent Writing logs 
[2019-09-12 06:52:06,366]  INFO - ServerManagement All server tasks have been completed.
[2019-09-12 06:52:06,366]  INFO - CarbonServerManager Shutting down WSO2 API Manager...
[2019-09-12 06:52:06,367]  INFO - CarbonServerManager Shutting down OSGi framework...
[2019-09-12 06:52:06,368]  INFO - CarbonServerManager Shutdown complete
[2019-09-12 06:52:06,368]  INFO - CarbonServerManager Halting JVM

Suggested Labels: helm, kubernetes

Suggested Assignees:

Affected Product Version: kubernetes-apim

OS, DB, other environment details and versions:
Azure AKS v1.13.9 Steps to reproduce: MySQL values.yaml by default apim-with-analytics values:

username: ""
password: ""
email: ""
cloudProvider: Azure

namespace: "wso2"
svcaccount: "wso2svc-account"
serverIp: ""
sharedDeploymentLocationPath: ""
sharedTenantsLocationPath: ""
helm install --name wso2apim-with-analytics-rdbms-service -f mysql/values.yaml stable/mysql --namespace wso2
helm install --name wso2-apim apim-with-analytics --namespace wso2
NAME:   wso2-apim
LAST DEPLOYED: Thu Sep 12 09:18:25 2019
NAMESPACE: wso2
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                         AGE
system:azure-cloud-provider  1s

==> v1/ClusterRoleBinding
NAME                         AGE
system:azure-cloud-provider  1s

==> v1/ConfigMap
NAME                        DATA  AGE
apim-analytics-conf-worker  1     1s
apim-conf                   4     1s
apim-conf-datasources       1     1s

==> v1/PersistentVolumeClaim
NAME                                                  STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
wso2apim-with-analytics-apim-deployment-volume-claim  Bound   pvc-83403030-d52d-11e9-8ff0-0a82c097e30c  1Gi       RWX           azurefile     1s
wso2apim-with-analytics-tenants-volume-claim          Bound   pvc-83393c93-d52d-11e9-8ff0-0a82c097e30c  1Gi       RWX           azurefile     1s

==> v1/Pod(related)
NAME                                                             READY  STATUS             RESTARTS  AGE
wso2apim-with-analytics-apim-analytics-deployment-7fdb6d987kt75  0/1    ContainerCreating  0         1s
wso2apim-with-analytics-apim-deployment-fcd674f6b-xzptj          0/1    ContainerCreating  0         1s

==> v1/Role
NAME                   AGE
endpoints-reader-role  1s

==> v1/Service
NAME                                            TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)                                                AGE
wso2apim-with-analytics-apim-analytics-service  ClusterIP  10.0.10.97    <none>       7612/TCP,7712/TCP,9444/TCP,9091/TCP,7071/TCP,7444/TCP  1s
wso2apim-with-analytics-apim-service            ClusterIP  10.0.164.103  <none>       8280/TCP,8243/TCP,9763/TCP,9443/TCP                    1s

==> v1/ServiceAccount
NAME             SECRETS  AGE
wso2svc-account  1        1s

==> v1/StorageClass
NAME       PROVISIONER               AGE
azurefile  kubernetes.io/azure-file  1s

==> v1beta1/Deployment
NAME                                               READY  UP-TO-DATE  AVAILABLE  AGE
wso2apim-with-analytics-apim-analytics-deployment  0/1    1           0          1s
wso2apim-with-analytics-apim-deployment            0/1    1           0          1s

==> v1beta1/Ingress
NAME                                  HOSTS                      ADDRESS  PORTS  AGE
wso2apim-with-analytics-apim-ingress  wso2apim,wso2apim-gateway  80, 443  1s

==> v1beta1/RoleBinding
NAME                                AGE
endpoints-reader-role-wso2-binding  1s

Related Issues:

Aaquiff commented 4 years ago

@brjlpz, this could be due to the delay in server startup. But since the liveness probe fails before server startup, there is a crash loop. Could you try increasing the liveness probe delay and doing a fresh install?