search

eclipse / hawkbit

Eclipse hawkBit™
https://projects.eclipse.org/projects/iot.hawkbit
Eclipse Public License 2.0
445 stars 184 forks source link

Hawkbit server crash on scheduled task #1674

Open hootbike opened 4 months ago

hootbike commented 4 months ago

Hello,

I have a hawkbit server behin traefik proxy, here the docker-compose config

mysql:
  image: "mysql:latest"
  environment:
    MYSQL_DATABASE: "hawkbit"
    MYSQL_ALLOW_EMPTY_PASSWORD: "true"
  restart: always
  ports:
      - "3306:3306"
  volumes:
    mysql-db-data:/var/lib/mysql
  labels:
    NAME: "mysql"
  networks:
    hawkbit-network

hawkbit:
  image: "hawkbit/hawkbit-update-server:latest-mysql"
  environment:
      - 'SPRING_DATASOURCE_URL=jdbc:mariadb://mysql:3306/hawkbit'
      - 'SPRING_RABBITMQ_HOST=rabbitmq'
      - 'SPRING_RABBITMQ_USERNAME=guest'
      - 'SPRING_RABBITMQ_PASSWORD=guest'
      - 'SPRING_DATASOURCE_USERNAME=root'
  volumes:
    artifactrepo:/artifactrepo
  restart: always
  ports:
      - "8080:8080"
  depends_on:
    mysql
  labels:
    NAME: "hawkbit"
    traefik.enable: true
    traefik.http.routers.hawkbit.rule: Host(`hawkbit.domain`)
    traefik.http.routers.hawkbit.entrypoints: hawkbit
    traefik.docker.network: docker-files_traefik-network
    traefik.http.services.hawkbit.loadbalancer.server.port: 8080
  networks:
    hawkbit-network
    traefik-network

volumes:
  mysql-db-data:
    driver: local
  artifactrepo:
    driver: local

networks:
  traefik-network:
  hawkbit-network:

It's work well for few hours, I can login, change and save parameters etc...

But after there is a crash:

2024-03-04 17:26:52.547  WARN 1 --- [executor-pool-1] o.m.jdbc.message.server.ErrorPacket      : Error: 1146-42S02: Table 'hawkbit.sp_tenant' doesn't exist
2024-03-04 17:26:52.585 ERROR 1 --- [executor-pool-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

org.springframework.jdbc.BadSqlGrammarException: ; bad SQL grammar []; nested exception is java.sql.SQLSyntaxErrorException: (conn=43) Table 'hawkbit.sp_tenant' doesn't exist
        at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:101)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
        at org.eclipse.hawkbit.repository.jpa.HawkBitEclipseLinkJpaDialect.searchAndTranslateSqlException(HawkBitEclipseLinkJpaDialect.java:85)
        at org.eclipse.hawkbit.repository.jpa.HawkBitEclipseLinkJpaDialect.translateJpaSystemExceptionIfPossible(HawkBitEclipseLinkJpaDialect.java:73)
        at org.eclipse.hawkbit.repository.jpa.HawkBitEclipseLinkJpaDialect.translateExceptionIfPossible(HawkBitEclipseLinkJpaDialect.java:64)
        at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
        at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
        at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:243)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
        at jdk.proxy2/jdk.proxy2.$Proxy234.findAll(Unknown Source)
        at org.eclipse.hawkbit.repository.jpa.management.JpaSystemManagement.findTenants(JpaSystemManagement.java:252)
        at org.eclipse.hawkbit.repository.jpa.management.JpaSystemManagement.forEachTenant(JpaSystemManagement.java:363)
        at org.eclipse.hawkbit.repository.jpa.management.JpaSystemManagement$$FastClassBySpringCGLIB$$4ce5fa1a.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:162)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
        at org.eclipse.hawkbit.repository.jpa.management.JpaSystemManagement$$EnhancerBySpringCGLIB$$1be21873.forEachTenant(<generated>)
        at org.eclipse.hawkbit.repository.jpa.rollout.RolloutScheduler.lambda$runningRolloutScheduler$1(RolloutScheduler.java:73)
        at org.eclipse.hawkbit.security.SystemSecurityContext.lambda$runAsSystemAsTenant$0(SystemSecurityContext.java:106)
        at org.eclipse.hawkbit.security.SecurityContextTenantAware.runInContext(SecurityContextTenantAware.java:153)
        at org.eclipse.hawkbit.security.SecurityContextTenantAware.runAsTenant(SecurityContextTenantAware.java:111)
        at org.eclipse.hawkbit.security.SystemSecurityContext.runAsSystemAsTenant(SystemSecurityContext.java:103)
        at org.eclipse.hawkbit.security.SystemSecurityContext.runAsSystem(SystemSecurityContext.java:74)
        at org.eclipse.hawkbit.repository.jpa.rollout.RolloutScheduler.runningRolloutScheduler(RolloutScheduler.java:65)
        at jdk.internal.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:82)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLSyntaxErrorException: (conn=43) Table 'hawkbit.sp_tenant' doesn't exist
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:282)
        at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:370)
        at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:134)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:872)
        at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:811)
        at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:730)
        at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:654)
        at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:95)
        at org.mariadb.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:288)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:1024)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:660)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:567)
        at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2105)
        at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:603)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:275)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:261)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:345)
        at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:765)
        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2776)
        at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2729)
        at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:584)
        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1232)
        at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:913)
        at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1191)
        at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:485)
        at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1279)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:3023)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1898)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1880)
        at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1845)
        at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:262)
        at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:482)
        at org.springframework.data.jpa.repository.support.SimpleJpaRepository.readPage(SimpleJpaRepository.java:761)
        at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:517)
        at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:482)
        at jdk.internal.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:289)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:530)
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:286)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:640)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:76)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        ... 55 common frames omitted

2024-03-04 17:28:12.979  WARN 1 --- [onnection adder] o.m.jdbc.message.server.ErrorPacket      : Error: 1049-42000: Unknown database 'hawkbit'
2024-03-04 17:28:22.984  WARN 1 --- [onnection adder] o.m.jdbc.message.server.ErrorPacket      : Error: 1049-42000: Unknown database 'hawkbit'
2024-03-04 17:28:32.990  WARN 1 --- [onnection adder] o.m.jdbc.message.server.ErrorPacket      : Error: 1049-42000: Unknown database 'hawkbit'
2024-03-04 17:28:42.996  WARN 1 --- [onnection adder] o.m.jdbc.message.server.ErrorPacket      : Error: 1049-42000: Unknown database 'hawkbit'
2024-03-04 17:28:53.003  WARN 1 --- [onnection adder] o.m.jdbc.message.server.ErrorPacket      : Error: 1049-42000: Unknown database 'hawkbit'

And I can't fully recover until I delete sql volume and rebuild the docker compose file ( which redo the migration on database). After that it's work few hours and crash again