search

ververica / ververica-platform-playground

Instructions for getting started with Ververica Platform on minikube.
https://docs.ververica.com/getting_started/index.html
Apache License 2.0
89 stars 39 forks source link

vvp-ververica-platform deployment CrashLoopBackOff due to SQLite missing table error #79

Closed StanimirIliev closed 2 years ago

StanimirIliev commented 2 years ago

Setup

I am trying to run the playground into on-prem Kubernetes cluster. From release-2.6 with minimal helm values edit as follows: minio

persistence:
  storageClass: directpv-min-io
  size: 100Gi

vvp

acceptCommunityEditionLicense: true
persistentVolume:
  storageClass: directpv-min-io

Then I am executing setup.sh

Error

The appmanager container on vvp-ververica-platform* pod keeps crashing due to failing to execute the following SQL query SELECT name, collection, value FROM appmanager WHERE collection = ? with the following exception org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager). This seems weird because I have logged on the kubernetes cluster host and can see that this table is there. Full container log

Here is the check that I do on the host:

[root@nuc-71032 pvc-81f069f7-af47-4819-806b-cedfbabc95ff]# ls
vvp.db  vvp.db-shm  vvp.db-wal
[root@nuc-71032 pvc-81f069f7-af47-4819-806b-cedfbabc95ff]# sqlite3 vvp.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> .tables
DATABASECHANGELOG      catalog_databases      gateway              
DATABASECHANGELOGLOCK  catalog_functions    
appmanager             catalog_tables       
sqlite> SELECT name, collection, value FROM appmanager WHERE collection = ?;
sqlite> SELECT name FROM appmanager;
575d4a23-85f6-4c93-9c45-da49c2213d4a
00000000-0000-0000-0000-000000000000
sqlite>

Please tell me if I need to provide additional details for this. Looking up to fix/workaround.

qinjunjerry commented 2 years ago

Normally, SQL error or missing database (no such table: appmanager) happens only the first time you set up VVP. Looking at the logs, it seems from the time when the error happened to the time when Received a poison pill. Terminating, there were only 200ms:

2022-03-05 18:25:05.248 ERROR 1 --- [           main] o.s.b.web.embedded.tomcat.TomcatStarter  : Error starting Tomcat context
...
2022-03-05 18:25:05.436   INFO 1 --- [lob-store-actor] c.d.a.p.i.blobservice.BlobStoreActor     : Received a poison pill. Terminating ...

Did you touch anything related to initialDelaySeconds (default to 90) or failureThreshold (default to 10) in the helm chart or any time related thing forced by your K8s?

StanimirIliev commented 2 years ago

Nothing more than the helm values overrides listed above. So the configurations you are asking about should be with their default values. Can you please elaborate what do you mean by "time related thing forced by K8s"?

qinjunjerry commented 2 years ago

Could you share the output of kubectl describe pod <vvp-pod> after the first crash? Also, does the appmanager container come up when you kill it and let K8s restart it?

StanimirIliev commented 2 years ago

Thank you for your help! Turns out there was another error after one that I mentioned - OOM I increased the memory to appmanager and gateway from 1Gi to 2Gi and now the whole deployment is up. By the way it is a little strange and misleading that even with the right resource configurations it fails one or two times due to the SQL error and on the third time it manages to start all three containers.

imalik8088 commented 2 years ago

I'm also facing the issue I've updated the resources from 1G 👉️ 2G but its not working. 

vvp-ververica-platform-5fdf86bf-v6t2t appmanager Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
vvp-ververica-platform-5fdf86bf-v6t2t appmanager 2022-07-07 07:40:58.368 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed
vvp-ververica-platform-5fdf86bf-v6t2t appmanager
vvp-ververica-platform-5fdf86bf-v6t2t appmanager org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) ~[spring-context-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:780) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:453) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.SpringApplication.run(SpringApplication.java:343) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1370) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1359) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at com.dataartisans.appmanager.apiserver.AppManagerApplication.main(AppManagerApplication.java:10) ~[app/:na]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:456) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:204) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    ... 8 common frames omitted
vvp-ververica-platform-5fdf86bf-v6t2t appmanager Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'proxyConfig': Unsatisfied dependency expressed through field 'sslKeyService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sslKeyService' defined in class path resource [com/dataartisans/appmanager/controller/core/modules/ServicesModule.class]: Unsatisfied dependency expressed through method 'sslKeyService' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jobManagerEndpointServiceFacade' defined in URL [jar:file:/vvp/app/lib/appmanager-controller-2.7.0-plain.jar!/com/dataartisans/appmanager/controller/core/JobManagerEndpointServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getPersistenceServiceForJob' defined in class path resource [com/dataartisans/appmanager/controller/core/modules/PersistenceModule.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.dataartisans.appmanager.controller.api.persistence.PersistenceService]: Factory method 'getPersistenceServiceForJob' threw exception; nested exception is java.util.concurrent.CompletionException: org.springframework.jdbc.UncategorizedSQLException:
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### Error querying database.  Cause: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager)
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### SQL: SELECT name, collection, value FROM appmanager WHERE collection = ?
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### Cause: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager)
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ; uncategorized SQLException; SQL state [null]; error code [1]; [SQLITE_ERROR] SQL error or missing database (no such table: appmanager); nested exception is org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager)
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:659) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:410) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) ~[spring-beans-5.3.20.jar:5.3.20]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:203) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:97) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:86) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:260) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5219) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[na:na]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[na:na]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:919) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.StandardService.startInternal(StandardService.java:432) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:927) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486) ~[tomcat-embed-core-9.0.63.jar:9.0.63]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) ~[spring-boot-2.5.14.jar:2.5.14]
vvp-ververica-platform-5fdf86bf-v6t2t appmanager    ... 13 common frames omitted
vvp-ververica-platform-5fdf86bf-v6t2t appmanager Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sslKeyService' defined in class path resource [com/dataartisans/appmanager/controller/core/modules/ServicesModule.class]: Unsatisfied dependency expressed through method 'sslKeyService' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jobManagerEndpointServiceFacade' defined in URL [jar:file:/vvp/app/lib/appmanager-controller-2.7.0-plain.jar!/com/dataartisans/appmanager/controller/core/JobManagerEndpointServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getPersistenceServiceForJob' defined in class path resource [com/dataartisans/appmanager/controller/core/modules/PersistenceModule.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.dataartisans.appmanager.controller.api.persistence.PersistenceService]: Factory method 'getPersistenceServiceForJob' threw exception; nested exception is java.util.concurrent.CompletionException: org.springframework.jdbc.UncategorizedSQLException:
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### Error querying database.  Cause: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager)
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### SQL: SELECT name, collection, value FROM appmanager WHERE collection = ?
vvp-ververica-platform-5fdf86bf-v6t2t appmanager ### Cause: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such table: appmanager)

Why its not working in the default setup?