search

gislab-augsburg / helm-charts

it@M Helm Charts
https://artifacthub.io/packages/search?repo=it-at-m
MIT License
0 stars 0 forks source link

dave-backend: Crash, Ressource Quota? #13

Closed gislab-augsburg closed 3 months ago

gislab-augsburg commented 3 months ago

dave-backend pod crashes after:

The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
  - Change member config using Java API: config.getJetConfig().setEnabled(true)
  - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
``` mb@nbo00370518:~/dave2/helm-charts/charts/dave$ oc logs dave-backend-service-7cd79c6b7d-6vbct -f Starting the Java application using /opt/jboss/container/java/run/run-java.sh ... INFO exec -a "java" java -Djavax.net.ssl.trustStore=/mnt/cacerts-lhm -Djavax.net.ssl.trustStorePassword=changeit -Xmx1g --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED" -Dhz.integritychecker.enabled=false -Dhz.jet.enabled=true -cp "." -jar /deployments/spring-boot-application.jar INFO running in /deployments Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhz.integritychecker.enabled=false -Dhz.jet.enabled=true WARNING: Unknown module: ALL-UNNAMED" specified to --add-opens --------------------------------------------------------------------------------------------------------------------------------------------------- _____ __ __ | __ \ /\ \ \ / / | | | | / \ \ \ / / ___ | | | | / /\ \ \ \/ / / _ \ | |__| | _ / ____ \ _ \ / | __/ _ |_____/ (_) /_/ \_\ (_) \/ \___| (_) _________________________ /\\ _____ | | | | | | \ ,-----, / \\____/__|__\_ |___|_____|_____|____|_|___\ ,--'---:---`--, / | _ | `| | | | \ ==(o)-----(o)==J `(o)-------(o)= --(o)(o)--------------(o)--' Datenbank und Auswertungen von Verkehrszählungen Application Name : dave-backend Spring Boot Version : (v2.7.8) --------------------------------------------------------------------------------------------------------------------------------------------------- 2024.03.13 11:25:24.006 | dave-backend | TraceId: | SpanId: ] | INFO | [background-preinit] | Version | [Version.java : 21] - HV000001: Hibernate Validator 6.2.5.Final 2024.03.13 11:25:24.080 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | DaveBackendApplication | [StartupInfoLogger.java : 55] - Starting DaveBackendApplication using Java 11.0.22 on dave-backend-service-7cd79c6b7d-6vbct with PID 1 (/deployments/spring-boot-application.jar started by 1003100000 in /deployments) 2024.03.13 11:25:24.081 | dave-backend | TraceId: | SpanId: ] | DEBUG | [main] | DaveBackendApplication | [StartupInfoLogger.java : 56] - Running with Spring Boot v2.7.8, Spring v5.3.25 2024.03.13 11:25:24.082 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | DaveBackendApplication | [SpringApplication.java : 637] - The following 1 profile is active: "dev" 2024.03.13 11:25:27.500 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 262] - Multiple Spring Data modules found, entering strict repository configuration mode 2024.03.13 11:25:27.501 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 132] - Bootstrapping Spring Data Elasticsearch repositories in DEFAULT mode. 2024.03.13 11:25:27.600 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 201] - Finished Spring Data repository scanning in 91 ms. Found 2 Elasticsearch repository interfaces. 2024.03.13 11:25:27.618 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 262] - Multiple Spring Data modules found, entering strict repository configuration mode 2024.03.13 11:25:27.619 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 132] - Bootstrapping Spring Data JPA repositories in DEFAULT mode. 2024.03.13 11:25:27.736 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data JPA - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.elasticsearch.CustomSuggestIndex; If you want this repository to be a JPA repository, consider annotating your entities with one of these annotations: javax.persistence.Entity, javax.persistence.MappedSuperclass (preferred), or consider extending one of the following types with your repository: org.springframework.data.jpa.repository.JpaRepository 2024.03.13 11:25:27.739 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data JPA - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.elasticsearch.ZaehlstelleIndex; If you want this repository to be a JPA repository, consider annotating your entities with one of these annotations: javax.persistence.Entity, javax.persistence.MappedSuperclass (preferred), or consider extending one of the following types with your repository: org.springframework.data.jpa.repository.JpaRepository 2024.03.13 11:25:27.763 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 201] - Finished Spring Data repository scanning in 136 ms. Found 7 JPA repository interfaces. 2024.03.13 11:25:28.619 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 262] - Multiple Spring Data modules found, entering strict repository configuration mode 2024.03.13 11:25:28.620 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 132] - Bootstrapping Spring Data Reactive Elasticsearch repositories in DEFAULT mode. 2024.03.13 11:25:28.640 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.InfoMessageRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.642 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.HochrechnungsfaktorRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.644 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.ChatMessageRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.645 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.PkwEinheitRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.647 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.ZeitintervallRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.660 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.EmailAddressRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.661 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationExtensionSupport | [RepositoryConfigurationExtensionSupport.java : 349] - Spring Data Reactive Elasticsearch - Could not safely identify store assignment for repository candidate interface de.muenchen.dave.repositories.relationaldb.DienstleisterRepository; If you want this repository to be a Reactive Elasticsearch repository, consider annotating your entities with one of these annotations: org.springframework.data.elasticsearch.annotations.Document (preferred), or consider extending one of the following types with your repository: org.springframework.data.elasticsearch.repository.ReactiveElasticsearchRepository 2024.03.13 11:25:28.662 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | RepositoryConfigurationDelegate | [RepositoryConfigurationDelegate.java : 201] - Finished Spring Data repository scanning in 41 ms. Found 0 Reactive Elasticsearch repository interfaces. 2024.03.13 11:25:29.775 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | GenericScope | [GenericScope.java : 283] - BeanFactory id=72baf491-10d4-373c-84b1-64d8a8b5f721 2024.03.13 11:25:32.222 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | TomcatWebServer | [TomcatWebServer.java : 108] - Tomcat initialized with port(s): 8080 (http) 2024.03.13 11:25:32.249 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | Http11NioProtocol | [DirectJDKLog.java : 173] - Initializing ProtocolHandler ["http-nio-8080"] 2024.03.13 11:25:32.250 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | StandardService | [DirectJDKLog.java : 173] - Starting service [Tomcat] 2024.03.13 11:25:32.251 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | StandardEngine | [DirectJDKLog.java : 173] - Starting Servlet engine: [Apache Tomcat/9.0.71] 2024.03.13 11:25:32.508 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | [/] | [DirectJDKLog.java : 173] - Initializing Spring embedded WebApplicationContext 2024.03.13 11:25:32.509 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | ServletWebServerApplicationContext | [ServletWebServerApplicationContext.java : 292] - Root WebApplicationContext: initialization completed in 8241 ms 2024.03.13 11:25:33.956 | dave-backend | TraceId: | SpanId: ] | WARN | [main] | HazelcastInstanceFactory | [StandardLoggerFactory.java : 56] - Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used: --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED 2024.03.13 11:25:34.206 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | logo | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] + + o o o o---o o----o o o---o o o----o o--o--o + + + + | | / \ / | | / / \ | | + + + + + o----o o o o o----o | o o o o----o | + + + + | | / \ / | | \ / \ | | + + o o o o o---o o----o o----o o---o o o o----o o 2024.03.13 11:25:34.207 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Copyright (c) 2008-2022, Hazelcast, Inc. All Rights Reserved. 2024.03.13 11:25:34.208 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Hazelcast Platform 5.1.5 (20221114 - c52d6ff) starting at [10.138.49.45]:5701 2024.03.13 11:25:34.208 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Cluster name: dave-backend-service-hazl_group 2024.03.13 11:25:34.209 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Starting Integrity Check scan. This is a costly operation and it can be disabled if startup time is important. To disable Integrity Checker do one of the following: - Change member config using Java API: config.setIntegrityCheckerEnabled(false); - Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to false - Add system property: -Dhz.integritychecker.enabled=false (for Hazelcast embedded, works only when loading config via Config.load) - Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=false (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) 2024.03.13 11:27:20.413 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Integrity Check scan finished in 104266 milliseconds 2024.03.13 11:32:47.468 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] Integrity Check finished in 432468 milliseconds 2024.03.13 11:33:13.265 | dave-backend | TraceId: | SpanId: ] | INFO | [main] | system | [StandardLoggerFactory.java : 56] - [10.138.49.45]:5701 [dave-backend-service-hazl_group] [5.1.5] The Jet engine is disabled. To enable the Jet engine on the members, do one of the following: - Change member config using Java API: config.getJetConfig().setEnabled(true) - Change XML/YAML configuration property: Set hazelcast.jet.enabled to true - Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load) - Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load) ```

CPU Quota: capture_001_13032024_120356

gislab-augsburg commented 3 months ago

Keine Fehlermeldung, warum der crasht. CPU ist hochgeschraubt auf 2, die EnvVars von dave-internal sind für backend drin: https://github.com/gislab-augsburg/helm-charts/blob/cf896adc7b54e0571254c3baa1249ca03991afc9/charts/dave/values.yaml#L54-L60

Die EnvVars von dave-c auch: https://github.com/gislab-augsburg/helm-charts/blob/cf896adc7b54e0571254c3baa1249ca03991afc9/charts/dave/values-lhm.yaml#L50-L57

Hab auch versucht, den jet engine zu enablen, bei JAVA_OPTIONS und JAVA_TOOL_OPTIONS mit -Dhz.jet.enabled=true und mit den EnvVars HZ_JET_ENABLED: true und hazelcast.jet.enabled: true, hat nicht geklappt.

@DanielOber kannst du helfen?

gislab-augsburg commented 3 months ago

Thanks to @klml we know now it was a memory error and a typo in postgresql port:

-> 43fb708c87e7f86abc815d4b6a49cd8a09b2a541, ddf689f3e6bb64163b2594417dcb020300c00b5b