Kubernetes apim Pod restarting (pattern 1) due to ERROR - RegistryContext Unable to get instance of the registry context

abhi1git commented 4 years ago

Description: I am installing latest helm chart of wso2 3.2.0 advanced pattern 1. When .Values.wso2.deployment.persistentRuntimeArtifacts.apacheSolrIndexing.enabled =false the apim deployment start correctly but when its value is made true (need to persist carbon data) then I get following logs and pod keeps on restarting.

Start WSO2 Carbon server
JAVA_HOME environment variable is set to /opt/java/openjdk
CARBON_HOME environment variable is set to /home/wso2carbon/wso2am-3.2.0
Using Java memory options: -Xms1024m -Xmx1024m
[2020-09-01 13:02:59,309]  INFO {org.wso2.config.mapper.ConfigParser} - Applying Configurations upon new Templates
[2020-09-01 13:02:59,311]  WARN {org.wso2.config.mapper.ConfigParser} - Overriding files in configuration directory /home/wso2carbon/wso2am-3.2.0
[2020-09-01 13:03:00,305]  INFO {org.wso2.config.mapper.ConfigParser} - Writing Metadata Entries...
andes [org.wso2.andes.wso2.internal.QpidBundleActivator] INFO : Setting BundleContext in PluginManager
callhome [org.wso2.carbon.callhome.internal.CallHomeActivator] DEBUG : Activating CallHome agent
[2020-09-01 13:03:12,512]  INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2020-09-01 13:03:12,521]  INFO - CarbonCoreActivator Operating System : Linux 4.9.0-11-amd64, amd64
[2020-09-01 13:03:12,521]  INFO - CarbonCoreActivator Java Home        : /opt/java/openjdk
[2020-09-01 13:03:12,522]  INFO - CarbonCoreActivator Java Version     : 11.0.8
[2020-09-01 13:03:12,522]  INFO - CarbonCoreActivator Java VM          : OpenJDK 64-Bit Server VM 11.0.8+10,AdoptOpenJDK
[2020-09-01 13:03:12,522]  INFO - CarbonCoreActivator Carbon Home      : /home/wso2carbon/wso2am-3.2.0
[2020-09-01 13:03:12,522]  INFO - CarbonCoreActivator Java Temp Dir    : /home/wso2carbon/wso2am-3.2.0/tmp
[2020-09-01 13:03:12,522]  INFO - CarbonCoreActivator User             : wso2carbon, en-US, GMT
[2020-09-01 13:03:12,785]  INFO - DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' won't be registered as an internal crypto provider. Please set the secret if the provider needs to be registered.
[2020-09-01 13:03:13,305]  INFO - KafkaEventAdapterServiceDS Successfully deployed the Kafka output event adaptor service
[2020-09-01 13:03:13,607]  INFO - TemplateDeployerServiceTrackerDS Successfully deployed the execution manager tracker service
[2020-09-01 13:03:17,392]  INFO - ConsentManagerComponent ConsentManagerComponent is activated.
[2020-09-01 13:03:17,605]  INFO - BinaryDataReceiver Started Binary SSL Transport on port : 9711
[2020-09-01 13:03:17,606]  INFO - BinaryDataReceiver Started Binary TCP Transport on port : 9611
[2020-09-01 13:03:17,679]  INFO - DataBridgeDS Successfully deployed Agent Server 
[2020-09-01 13:03:20,920] ERROR - RegistryContext Unable to get instance of the registry context
org.wso2.carbon.registry.core.exceptions.RegistryException: Unable to connect to Data Source
    at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:165) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryContext.<init>(RegistryContext.java:462) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryContext.<init>(RegistryContext.java:443) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryContext.getBaseInstance(RegistryContext.java:322) [org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.getEmbeddedRegistryService(RegistryCoreServiceComponent.java:587) [org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.buildRegistryService(RegistryCoreServiceComponent.java:520) [org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.activate(RegistryCoreServiceComponent.java:123) [org.wso2.carbon.registry.core_4.6.0.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:73) [org.wso2.carbon.user.core_4.6.0.jar:?]
    at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61) [org.wso2.carbon.user.core_4.6.0.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:842) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) [?:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:834) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:791) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1013) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.Module.doStart(Module.java:598) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.Module.start(Module.java:462) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$1.run(ModuleContainer.java:1820) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$2$1.execute(EquinoxContainerAdaptor.java:150) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1813) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1770) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1735) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1661) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
Caused by: org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: null [90028-199]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.message.DbException.get(DbException.java:194) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.convertIllegalStateException(MVTableEngine.java:197) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:173) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:95) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.getPageStore(Database.java:2739) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.open(Database.java:769) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.<init>(Database.java:313) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.Driver.connect(Driver.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:739) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:671) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:150) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    ... 46 more
Caused by: java.lang.IllegalStateException: Could not open file nio:/home/wso2carbon/solr/database/WSO2CARBON_DB.mv.db [1.4.199/1]
    at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.FileStore.open(FileStore.java:179) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.MVStore.<init>(MVStore.java:390) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.MVStore$Builder.open(MVStore.java:3343) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:162) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:95) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.getPageStore(Database.java:2739) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.open(Database.java:769) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.<init>(Database.java:313) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.Driver.connect(Driver.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:739) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:671) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:150) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    ... 46 more
Caused by: java.io.IOException: No locks available
    at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) ~[?:?]
    at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:96) ~[?:?]
    at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1161) ~[?:?]
    at org.h2.store.fs.FileNio.tryLock(FilePathNio.java:121) ~[h2_1.4.199.wso2v1.jar:?]
    at java.nio.channels.FileChannel.tryLock(FileChannel.java:1165) ~[?:?]
    at org.h2.mvstore.FileStore.open(FileStore.java:163) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.MVStore.<init>(MVStore.java:390) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.MVStore$Builder.open(MVStore.java:3343) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine$Store.open(MVTableEngine.java:162) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.mvstore.db.MVTableEngine.init(MVTableEngine.java:95) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.getPageStore(Database.java:2739) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.open(Database.java:769) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.openDatabase(Database.java:319) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Database.<init>(Database.java:313) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.openSession(Engine.java:201) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:161) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.Engine.createSession(Engine.java:31) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:336) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148) ~[h2_1.4.199.wso2v1.jar:?]
    at org.h2.Driver.connect(Driver.java:69) ~[h2_1.4.199.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:739) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:671) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
    at org.wso2.carbon.registry.core.config.RegistryConfigurationProcessor.populateRegistryConfig(RegistryConfigurationProcessor.java:150) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    ... 46 more
[2020-09-01 13:03:20,930] ERROR - RegistryCoreServiceComponent Failed to activate Registry Core bundle 
java.lang.NullPointerException: null
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.getEmbeddedRegistryService(RegistryCoreServiceComponent.java:588) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.buildRegistryService(RegistryCoreServiceComponent.java:520) ~[org.wso2.carbon.registry.core_4.6.0.jar:?]
    at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.activate(RegistryCoreServiceComponent.java:123) [org.wso2.carbon.registry.core_4.6.0.jar:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:73) [org.wso2.carbon.user.core_4.6.0.jar:?]
    at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61) [org.wso2.carbon.user.core_4.6.0.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:842) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) [?:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:834) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:791) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1013) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.Module.doStart(Module.java:598) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.Module.start(Module.java:462) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$1.run(ModuleContainer.java:1820) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$2$1.execute(EquinoxContainerAdaptor.java:150) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1813) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1770) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1735) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1661) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]

Affected Product Version: 3.2.0/Advanced/Pattern 1 OS, DB, other environment details and versions:
Kubernetes(v1.16.9) Cluster 3 master and 2 nodes Steps to reproduce: Change apacheSolrIndexing.enabled =true in values.yaml file of pattern1 an then install helm chart

chirangaalwis commented 4 years ago

Please refer to https://github.com/wso2/kubernetes-apim/issues/397#issuecomment-660429579 for suggested solutions for this issue.

Also, when switching to persistence for Apache based Solr-Indexing, we recommend you to perform a clean deployment (i.e. to delete any Pods remaining from a previous deployment of this pattern, if any using a helm uninstall) using helm install rather than using a Helm based upgrade (i.e. using the helm upgrade command) on an existing deployment.

chirangaalwis commented 4 years ago

@abhi1git in connection with the https://github.com/wso2/kubernetes-apim/issues/457#issuecomment-684952601, for detailed information on switching to a different persistent storage solution, please refer to this section of the documentation.

abhi1git commented 4 years ago

Please refer to #397 (comment) for suggested solutions for this issue.

Also, when switching to persistence for Apache based Solr-Indexing, we recommend you to perform a clean deployment (i.e. to delete any Pods remaining from a previous deployment of this pattern, if any using a helm uninstall) using helm install rather than using a Helm based upgrade (i.e. using the helm upgrade command) on an existing deployment.

Yes I have already enabled NFS server provisioner and using storageclass nfs only for all wso2 pvc claims also had done a clean deployment when enabled solr indexing still facing this issue and then raised it here. Please let me know if there's a fix available for this or if it is going to be considered for next release

Also wanted to suggest if helm charts can be developed to persist full home directory of wso2 so that any configuration changes I do in other config files for example identity.xml etc won't be lost.

Thanks @chirangaalwis Awaiting your reply.

chirangaalwis commented 4 years ago

@abhi1git can you please elaborate your use case? In the long run, how many APIs do you plan manage via WSO2 API Manager?

Yes I have already enabled NFS server provisioner and using storageclass nfs only for all wso2 pvc claims also had done a clean deployment when enabled solr indexing still facing this issue and then raised it here.

Did you install the NFS Server Provisioner based Kubernetes StorageClass via the WSO2 product Helm chart (i.e. in the form of a dependency of WSO2 product Helm chart)? Or did you perform an independent, cluster wide installation of the NFS Server Provisioner?

Can you explain the exact steps you have followed when performing the clean deployment (including the Helm based commands) in which you encountered the discussed issue?

Furthermore, what is the infrastructure you have used to create the Kubernetes cluster (i.e. whether it is cloud-based or it is on bare metal, if cloud based what is the cloud vendor)?

Please let me know if there's a fix available for this or if it is going to be considered for next release

Also wanted to suggest if helm charts can be developed to persist full home directory of wso2 so that any configuration changes I do in other config files for example identity.xml etc won't be lost.

WSO2 recommends the use of Kubernetes ConfigMaps to introduce configuration changes to a deployment. Using a ConfigMap ensures, that you pass the same configuration change to the product container based Pods every time you spawn a new instance.

abhi1git commented 4 years ago

Infrastructure : Kubernetes cluster on AWS cloud created using kops
Installed NFS Server Provisioner based Kubernetes StorageClass via the WSO2 product Helm chart
When performed clean deployment (helm install) by deleting all existing resources the wso2 api manager pod starts as expected

As suggested by you tried passing configuration using configmaps (for changing axis2.xml and axis2_client.xml) then pod enters CrashLoopBackoff by giving following logs

[2020-09-07 15:06:10,196]  INFO {org.wso2.config.mapper.ConfigParser} - Applying Configurations upon new Templates
[2020-09-07 15:06:10,198]  WARN {org.wso2.config.mapper.ConfigParser} - Overriding files in configuration directory /home/wso2carbon/wso2am-3.2.0
[2020-09-07 15:06:11,122]  SEVERE {org.wso2.carbon.server.Main handleConfiguration} - Error while performing configuration changes 
org.wso2.config.mapper.ConfigParserException: Error while store new configurations
at org.wso2.config.mapper.ConfigParser.parse(ConfigParser.java:132)
at org.wso2.carbon.server.Main.handleConfiguration(Main.java:231)
at org.wso2.carbon.server.Main.main(Main.java:103)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.wso2.carbon.bootstrap.Bootstrap.loadClass(Bootstrap.java:70)
at org.wso2.carbon.bootstrap.Bootstrap.main(Bootstrap.java:51)
Caused by: java.io.IOException: Read-only file system
at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.base/java.io.File.createNewFile(File.java:1026)
at org.wso2.config.mapper.ConfigParser.deploy(ConfigParser.java:223)
at org.wso2.config.mapper.ConfigParser.deployAndStoreMetadata(ConfigParser.java:180)
at org.wso2.config.mapper.ConfigParser.parse(ConfigParser.java:127)
... 8 more

Also tried loading axis2 configmaps to wso2-config-volume which is the procedure followed for deployment.toml file symlink creation and the pod starts in correct manner but the config changes are not reflected in original files /home/wso2carbon/wso2am-3.2.0/repository/conf/axis2/axis2.xml

Can you please guide about or point me to an example which demonstrates passing configmaps for any other config file than deployment.toml. Or please elaborate on how changes from deployment.toml of wso2-config-volume are getting reflected in /home/wso2carbon/wso2am-3.2.0/repository/conf/deployment.toml

@chirangaalwis

wso2 / kubernetes-apim

Kubernetes apim Pod restarting (pattern 1) due to ERROR - RegistryContext Unable to get instance of the registry context #457