Closed genedev22 closed 1 week ago
alfespa17
commented
2 weeks ago Hello @genedev22 I am not sure why redis is failing, the helm chart is using the bitnami helm chart as you can see here.
Maybe you can try installing installing that redis helm chart from bitnami individually just to check is working or maybe you can use a newer version provided by bitnami
alfespa17
commented
2 weeks ago If you install redis using the bitnami helm chart you can see this pr that added support to use an external redis
genedev22
commented
1 week ago @alfespa17 deploying that Redis dependency chart was a good troubleshooting strategy. It allowed me to make partial progress on the errors in the API pod.
Running a standalone Redis chart allowed me to observe that similar PVCs (from both the standalone Redis chart and the Terrakube chart) were not getting fulfilled.
To resolve this, I installed/configured the AWS EBS CSI driver (to automatically provision storage space from the node's volume for PVCs) and configured the cluster to use a default storage class that's linked to this CSI driver so the Redis PVCs (which do not specify a storage class on my end) get fulfilled.
The PVC (all I see is one) is fulfilled for the default Redis pod.
$ k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
redis-data-terrakube-redis-master-0 Bound pvc-3a58bcb8-6ffa-45a4-9512-5cc51601279c 8Gi RWO gp3 <unset> 3h1mOn the API pod: The Jedis client in the API pod is not throwing an exception from Redis health checks right now (but there are DNS issues lingering in the API pod logs -- I'll get back to this in a moment).
Calculating JVM memory based on 7233400K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx6449367K -XX:MaxMetaspaceSize=272032K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 7233400K, Thread Count: 250, Loaded Class Count: 45614, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_opentelemetry/opentelemetry-java/opentelemetry-javaagent.jar -XX:MaxDirectMemorySize=10M -Xmx6449367K -XX:MaxMetaspaceSize=272032K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-09-26 16:55:29:011 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.0
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.1.12)
2024-09-26T16:55:31.042Z INFO 1 --- [ main] org.terrakube.api.ServerApplication : Starting ServerApplication v2.22.0 using Java 17.0.11 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2024-09-26T16:55:31.052Z INFO 1 --- [ main] org.terrakube.api.ServerApplication : The following 1 profile is active: "demo"
2024-09-26T16:55:33.450Z INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode
2024-09-26T16:55:33.450Z INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2024-09-26T16:55:33.754Z INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 291 ms. Found 18 JPA repository interfaces.
2024-09-26T16:55:35.485Z INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=520ea4d5-88d4-3dfc-8ce7-88472877d7e2
2024-09-26T16:55:37.015Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2024-09-26T16:55:37.034Z INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2024-09-26T16:55:37.035Z INFO 1 --- [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.24]
2024-09-26T16:55:37.234Z INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2024-09-26T16:55:37.238Z INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 5927 ms
2024-09-26T16:55:38.215Z INFO 1 --- [ main] o.t.a.p.s.StreamingConfiguration : Redis Configuration=> User: username is null, Hostname: terrakube-redis-master, Port: 6379, Ssl: false
2024-09-26T16:55:38.222Z INFO 1 --- [ main] o.t.a.p.s.StreamingConfiguration : Redis connection is not using username parameter
2024-09-26T16:55:38.222Z INFO 1 --- [ main] o.t.a.p.s.StreamingConfiguration : Using default Redis connection
2024-09-26T16:55:38.576Z INFO 1 --- [ main] o.t.a.p.d.DataSourceAutoConfiguration : DataSourceType: POSTGRESQL
2024-09-26T16:55:38.577Z INFO 1 --- [ main] o.t.a.p.d.DataSourceAutoConfiguration : postgresql datasource using SSL Mode: prefer
2024-09-26T16:55:42.405Z INFO 1 --- [ main] liquibase.changelog : Reading from public.databasechangelog
2024-09-26T16:55:42.970Z INFO 1 --- [ main] liquibase.ui : Database is up to date, no changesets to execute
2024-09-26T16:55:42.976Z INFO 1 --- [ main] liquibase.changelog : Reading from public.databasechangelog
2024-09-26T16:55:43.009Z INFO 1 --- [ main] liquibase.util : UPDATE SUMMARY
2024-09-26T16:55:43.009Z INFO 1 --- [ main] liquibase.util : Run: 0
2024-09-26T16:55:43.009Z INFO 1 --- [ main] liquibase.util : Previously run: 56
2024-09-26T16:55:43.010Z INFO 1 --- [ main] liquibase.util : Filtered out: 0
2024-09-26T16:55:43.010Z INFO 1 --- [ main] liquibase.util : -------------------------------
2024-09-26T16:55:43.010Z INFO 1 --- [ main] liquibase.util : Total change sets: 56
2024-09-26T16:55:43.011Z INFO 1 --- [ main] liquibase.util : Update summary generated
2024-09-26T16:55:43.052Z INFO 1 --- [ main] liquibase.lockservice : Successfully released change log lock
2024-09-26T16:55:43.056Z INFO 1 --- [ main] liquibase.command : Command execution complete
2024-09-26T16:55:43.486Z INFO 1 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2024-09-26T16:55:43.580Z INFO 1 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 6.2.25.Final
2024-09-26T16:55:43.584Z INFO 1 --- [ main] org.hibernate.cfg.Environment : HHH000406: Using bytecode reflection optimizer
2024-09-26T16:55:44.019Z INFO 1 --- [ main] o.s.o.j.p.SpringPersistenceUnitInfo : No LoadTimeWeaver setup: ignoring JPA class transformer
2024-09-26T16:55:44.469Z WARN 1 --- [ main] o.terrakube.api.rs.module.GitTagsCache : Module Config: MaxTotal 128 MaxIdle 128 MinIdle 64 Timeout 600000 Schedule 0 */3 * ? * *
2024-09-26T16:55:44.473Z WARN 1 --- [ main] o.terrakube.api.rs.module.GitTagsCache : Connecting Default Redis using hostname, port and password
2024-09-26T16:55:44.475Z INFO 1 --- [ main] o.terrakube.api.rs.module.GitTagsCache : Redis connection completed...
2024-09-26T16:55:48.307Z INFO 1 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)
2024-09-26T16:55:48.312Z INFO 1 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2024-09-26T16:55:49.449Z INFO 1 --- [ main] o.t.a.p.s.c.StorageTypeAutoConfiguration : StorageType=AWS
2024-09-26T16:55:51.581Z INFO 1 --- [ main] org.quartz.impl.StdSchedulerFactory : Using default implementation for ThreadExecutor
2024-09-26T16:55:51.617Z INFO 1 --- [ main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2024-09-26T16:55:51.617Z INFO 1 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v2.5.0-rc1 created.
2024-09-26T16:55:51.668Z INFO 1 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : Using db table-based data access locking (synchronization).
2024-09-26T16:55:51.672Z INFO 1 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : JobStoreCMT initialized.
2024-09-26T16:55:51.674Z INFO 1 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.5.0-rc1) 'schedulerFactoryBean' with instanceId 'terrakube-api-6f69c546cd-pwmh61727369751585'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.springframework.scheduling.quartz.LocalDataSourceJobStore' - which supports persistence. and is clustered.
2024-09-26T16:55:51.676Z INFO 1 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'schedulerFactoryBean' initialized from an externally provided properties instance.
2024-09-26T16:55:51.677Z INFO 1 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler version: 2.5.0-rc1
2024-09-26T16:55:51.677Z INFO 1 --- [ main] org.quartz.core.QuartzScheduler : JobFactory set to: org.terrakube.api.plugin.scheduler.configuration.QuartzAutoConfiguration$AutowireCapableBeanJobFactory@6571d5a1
2024-09-26T16:55:51.784Z INFO 1 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : ClusterManager: detected 1 failed or restarted instances.
2024-09-26T16:55:51.785Z INFO 1 --- [ main] o.s.s.quartz.LocalDataSourceJobStore : ClusterManager: Scanning for instance "terrakube-api-7874cd4b98-lrpqc1727364670087"'s failed in-progress jobs.
2024-09-26T16:55:51.798Z INFO 1 --- [ main] org.quartz.core.QuartzScheduler : Scheduler schedulerFactoryBean_$_terrakube-api-6f69c546cd-pwmh61727369751585 started.
2024-09-26T16:55:51.899Z INFO 1 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore : Handling 1 trigger(s) that missed their scheduled fire-time.
2024-09-26T16:55:52.308Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Run module index scan
2024-09-26T16:55:52.310Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Create Schedule Module Refresh: DEFAULT.TerrakubeV2_ModuleRefresh_7f83c210-6f94-4a28-8934-ebd5e8c79e96
2024-09-26T16:55:52.383Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Disable Old Module Refresh
2024-09-26T16:55:52.493Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : jobDetail is null false
2024-09-26T16:55:52.493Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Delete Old Quartz Job
2024-09-26T16:55:52.701Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Reschedule with new frequency
2024-09-26T16:55:52.702Z INFO 1 --- [ main] o.t.a.p.scheduler.module.CacheService : Create Schedule Job Trigger DEFAULT.TerrakubeV2_ModuleRefresh
2024-09-26T16:55:53.692Z WARN 1 --- [ main] .s.s.UserDetailsServiceAutoConfiguration :
Using generated security password: <snip>
This generated password is for development use only. Your security configuration must be updated before running your application in production.
2024-09-26T16:55:54.166Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/compute/azurerm
2024-09-26T16:55:54.166Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/compute/azurerm
2024-09-26T16:55:54.167Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.167Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.304Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/cosmosdb/azurerm
2024-09-26T16:55:54.304Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.316Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/cosmosdb/azurerm
2024-09-26T16:55:54.316Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.419Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/database/azurerm
2024-09-26T16:55:54.420Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.445Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/database/azurerm
2024-09-26T16:55:54.445Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:54.849Z INFO 1 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-09-26T16:55:55.033Z INFO 1 --- [ main] o.t.a.p.s.a.dex.DexWebSecurityAdapter : Loading CORS https://terrakube-ui.<snipped>.com
2024-09-26T16:55:55.071Z INFO 1 --- [ main] o.s.s.web.DefaultSecurityFilterChain : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@667dfd42, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1234d9f6, org.springframework.security.web.context.SecurityContextHolderFilter@6cde71fe, org.springframework.security.web.header.HeaderWriterFilter@443d8d4d, org.springframework.web.filter.CorsFilter@6e4c8583, org.springframework.security.web.csrf.CsrfFilter@7215172a, org.springframework.security.web.authentication.logout.LogoutFilter@469b55ae, org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter@44971269, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@deddccb, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@4e80eb4c, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@451a5843, org.springframework.security.web.access.ExceptionTranslationFilter@5e4c74ec, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@21f9c71e]
2024-09-26T16:55:55.577Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/manageddisk/azurerm
2024-09-26T16:55:55.577Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:55.579Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/manageddisk/azurerm
2024-09-26T16:55:55.579Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:56.729Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/network/azurerm
2024-09-26T16:55:56.730Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:55:56.729Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/network/azurerm
2024-09-26T16:55:56.733Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.
2024-09-26T16:56:05.897Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.time.Instant
2024-09-26T16:56:05.948Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.time.OffsetDateTime
2024-09-26T16:56:05.949Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.util.TimeZone
2024-09-26T16:56:05.950Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.net.URL
2024-09-26T16:56:05.951Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.util.Date
2024-09-26T16:56:05.951Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.sql.Date
2024-09-26T16:56:05.952Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.sql.Time
2024-09-26T16:56:05.952Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class java.sql.Timestamp
2024-09-26T16:56:05.963Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Day
2024-09-26T16:56:05.968Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Hour
2024-09-26T16:56:05.972Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.ISOWeek
2024-09-26T16:56:05.975Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Minute
2024-09-26T16:56:05.978Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Month
2024-09-26T16:56:05.981Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Quarter
2024-09-26T16:56:05.984Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Second
2024-09-26T16:56:05.986Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Time
2024-09-26T16:56:05.990Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Week
2024-09-26T16:56:05.996Z INFO 1 --- [ main] com.yahoo.elide.Elide : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Year
ANTLR Tool version 4.13.1 used for code generation does not match the current runtime version 4.10.1
ANTLR Runtime version 4.13.1 used for parser compilation does not match the current runtime version 4.10.1
ANTLR Tool version 4.13.1 used for code generation does not match the current runtime version 4.10.1
ANTLR Runtime version 4.13.1 used for parser compilation does not match the current runtime version 4.10.1
2024-09-26T16:56:07.872Z INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2024-09-26T16:56:08.131Z INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-09-26T16:56:08.137Z INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2024-09-26T16:56:08.150Z INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 13 ms
2024-09-26T16:56:08.967Z INFO 1 --- [ main] org.terrakube.api.ServerApplication : Started ServerApplication in 38.976 seconds (process running for 40.424)
Unfortunately, the executor pod is still throwing a Jedis exception. Here are the full logs.
Calculating JVM memory based on 7254968K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx6545616K -XX:MaxMetaspaceSize=197351K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 7254968K, Thread Count: 250, Loaded Class Count: 32429, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_opentelemetry/opentelemetry-java/opentelemetry-javaagent.jar -XX:MaxDirectMemorySize=10M -Xmx6545616K -XX:MaxMetaspaceSize=197351K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-09-26 16:55:35:020 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.0
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.1.12)
[main] INFO org.terrakube.executor.ExecutorApplication - Starting ExecutorApplication v2.22.0 using Java 17.0.11 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
[main] INFO org.terrakube.executor.ExecutorApplication - No active profile set, falling back to 1 default profile: "default"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Multiple Spring Data modules found, entering strict repository configuration mode
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Bootstrapping Spring Data Redis repositories in DEFAULT mode.
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 128 ms. Found 0 Redis repository interfaces.
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8090 (http)
Sep 26, 2024 4:55:39 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 26, 2024 4:55:39 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet engine: [Apache Tomcat/10.1.24]
Sep 26, 2024 4:55:39 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring embedded WebApplicationContext
[main] INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 2947 ms
[main] INFO org.terrakube.executor.configuration.RedisAutoConfiguration - Redis User: NULL username, Hostname: terrakube-redis-master, Port: 6379, Ssl: false
[main] INFO org.terrakube.executor.configuration.RedisAutoConfiguration - Redis connection with default configuration
[main] INFO org.terrakube.executor.configuration.LocalConfiguration - Output Enabled: AwsTerraformOutputImpl
[main] INFO org.terrakube.executor.configuration.LocalConfiguration - State Enabled: AwsTerraformStateImpl
[main] WARN org.springframework.boot.autoconfigure.groovy.template.GroovyTemplateAutoConfiguration - Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)
[main] INFO org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
[main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8090 (http) with context path ''
[main] INFO org.terrakube.executor.service.mode.batch.BatchModeServiceImpl - Ephemeral mode is enable: false
[main] INFO org.terrakube.executor.ExecutorApplication - Started ExecutorApplication in 8.204 seconds (process running for 9.152)
Sep 26, 2024 4:55:48 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring DispatcherServlet 'dispatcherServlet'
[http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
[http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 2 ms
[http-nio-8090-exec-1] WARN org.springframework.boot.actuate.data.redis.RedisHealthIndicator - Redis health check failed
org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:276)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:459)
at org.springframework.data.redis.core.RedisConnectionUtils.fetchConnection(RedisConnectionUtils.java:194)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:143)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:104)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
at org.springframework.boot.actuate.data.redis.RedisHealthIndicator.doHealthCheck(RedisHealthIndicator.java:49)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:94)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:47)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getLoggedHealth(HealthEndpointSupport.java:172)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:145)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateContribution(HealthEndpointSupport.java:156)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:141)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:110)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:81)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:80)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:69)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:281)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:327)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:434)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:731)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.
at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:63)
at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:87)
at redis.clients.jedis.Connection.connect(Connection.java:188)
at redis.clients.jedis.Connection.initializeFromClientConfig(Connection.java:374)
at redis.clients.jedis.Connection.<init>(Connection.java:61)
at redis.clients.jedis.Jedis.<init>(Jedis.java:214)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:178)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:298)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:223)
at redis.clients.jedis.util.Pool.getResource(Pool.java:38)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:378)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:17)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:267)
... 82 more
Suppressed: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:73)
... 95 more
Why the executor pod is continuing to have errors is not clear to me. This is why:
I can connect to the internal redis endpoint from another pod in the same cluster. To elaborate a little: I deployed a pod called redis-client. From there, redis-cli connects to the internal hostname terrakube-redis-master.
k run redis-client --restart='Never' --image docker.io/bitnami/redis:7.4.0-debian-12-r4 --command -- sleep infinity
k exec --tty -i redis-client -- /bin/bash
redis-cli -h terrakube-redis-master -p 6379 -a <password>If you can suggest where else to look, that would be appreciated. My guess is that I need to directly address the DNS errors still showing in the API pod; I'm hoping that will also resolve the executor pod's Redis error.
alfespa17
commented
1 week ago Hello @genedev22
It looks like the API pod cant resolve Github address for the samples modules that the API is loading.
2024-09-26T16:55:54.166Z ERROR 1 --- [ryBean_Worker-2] o.t.a.plugin.scheduler.module.CacheJob : Updating module index for azure/compute/azurerm
2024-09-26T16:55:54.167Z ERROR 1 --- [ryBean_Worker-1] o.t.a.plugin.scheduler.module.CacheJob : Failed to connect to any host resolved for DNS name.If you dont want to load the sample data change this in your helm values api.loadSampleData.
From what I can see the API is connecting correctly to Redis, the executor pod maybe is failing because of some DNS issue, maybe it is related to the same issue from above for the API
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.When you deployed using the default redis, the API is able to resolve redis using the following hostname, you can check the service name with kubectl.
The same happens inside the API.
So if the API is able to connect not sure why the executor is not able to resolve the internal kubernes dns service for Redis, did you try to delete the executor pod?
genedev22
commented
1 week ago Hi @alfespa17
Quick remark on the sample data: github.com is reachable from within the API pod; I modified the Helm chart to install another container in the API pod and from there was able to run ping and wget from against github.com.
On the Executor pod: Surprisingly, yes, deleting the Executor pod (and letting it re-spawn) did lead to the absence of the Redis health check error / Jedis stack trace. Here are the new Executor pod logs:
$ k logs terrakube-executor-7769fb89ff-2bf57
Calculating JVM memory based on 6867196K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx6157844K -XX:MaxMetaspaceSize=197351K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 6867196K, Thread Count: 250, Loaded Class Count: 32429, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_opentelemetry/opentelemetry-java/opentelemetry-javaagent.jar -XX:MaxDirectMemorySize=10M -Xmx6157844K -XX:MaxMetaspaceSize=197351K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-09-28 19:10:21:883 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.0
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.1.12)
[main] INFO org.terrakube.executor.ExecutorApplication - Starting ExecutorApplication v2.22.0 using Java 17.0.11 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
[main] INFO org.terrakube.executor.ExecutorApplication - No active profile set, falling back to 1 default profile: "default"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Multiple Spring Data modules found, entering strict repository configuration mode
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Bootstrapping Spring Data Redis repositories in DEFAULT mode.
[main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 208 ms. Found 0 Redis repository interfaces.
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.core.io.support.PropertySourceProcessor - Properties location [classpath:application-${spring.profiles.active}.properties] not resolvable: Could not resolve placeholder 'spring.profiles.active' in value "classpath:application-${spring.profiles.active}.properties"
[main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8090 (http)
Sep 28, 2024 7:10:27 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Sep 28, 2024 7:10:27 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet engine: [Apache Tomcat/10.1.24]
Sep 28, 2024 7:10:27 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring embedded WebApplicationContext
[main] INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 3099 ms
[main] INFO org.terrakube.executor.configuration.RedisAutoConfiguration - Redis User: NULL username, Hostname: terrakube-redis-master, Port: 6379, Ssl: false
[main] INFO org.terrakube.executor.configuration.RedisAutoConfiguration - Redis connection with default configuration
[main] INFO org.terrakube.executor.configuration.LocalConfiguration - Output Enabled: AwsTerraformOutputImpl
[main] INFO org.terrakube.executor.configuration.LocalConfiguration - State Enabled: AwsTerraformStateImpl
[main] WARN org.springframework.boot.autoconfigure.groovy.template.GroovyTemplateAutoConfiguration - Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)
[main] INFO org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 1 endpoint(s) beneath base path '/actuator'
[main] INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8090 (http) with context path ''
[main] INFO org.terrakube.executor.service.mode.batch.BatchModeServiceImpl - Ephemeral mode is enable: false
[main] INFO org.terrakube.executor.ExecutorApplication - Started ExecutorApplication in 8.817 seconds (process running for 9.799)
Sep 28, 2024 7:10:41 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring DispatcherServlet 'dispatcherServlet'
[http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcher
[http-nio-8090-exec-1] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 5 msI had to delete the Executor pod after each helm install to get its Redis errors to disappear.
I'm working on associating DNS entries (defined in Route53) to an application load balancer (ALB) that reaches the EKS cluster. This need to delete the Executor pod after each Helm install (to stop getting Redis errors) might go away after I workout my DNS issues. I'll have to see.
Since I feel the Redis issue has been essentially addressed, then I'm OK with closing this issue for now; I can drop a note if I see that fixing my DNS issues correlates to the Executor pod initializing successfully on its first try. Thanks again.
genedev22
commented
1 day ago Update: I managed to get my DNS and load balancer issues resolved and I can now log into the Terrakube application, but the Executor pod still has to be deleted/re-spawned after running helm install ... after the Jedis exception. The reason for this is still undetermined by me.
alfespa17
commented
1 day ago hello @genedev22 maybe the REDIS pod is not available before the executor pod completes its own initialization, that is why when you kill the executor pods and starts again it can connect to REDIS successfully, but even if you see that error in the executor when it is running a job I think it should work correctly.
You can try sending one job just to test
Hi there. I'm interested in using Terrakube in my AWS account under an EKS cluster.
I'm trying to follow the AWS example from the
terrakube-helm-chartrepo: https://github.com/AzBuilder/terrakube-helm-chart/blob/main/examples/CognitoAuthentication-Example1.mdvalues.yamlfile follows the values schema in theterrakube-3.21.0Helm chart.I have all my AWS resources provisioned and ran
helm install. Unfortunately, I keep getting a Redis health check failure in the API and Executor pods.terrakubenamespace).Right now I'm trying to make the default Redis resource work. The Terrakube pods eventually reach this status (the API and Executor pods keep restarting and the Redis pod doesn't come online).
When looking at the API pod's logs, I get loops of: Redis health check fails, an exception with the Jedis client, and a long stack trace. A similar pattern occurs for the Executor pod logs.
The API logs are a little different, though. Before the cycles of health check failures start, I am seeing various DNS errors.
Note: The Postgres DB (RDS Postgres DB via SSL in my case) appears to be connectable from the API pod according to the logs. I further confirmed that I can connect to it myself from a Postgres CLI client from within the cluster (in the
terrakubenamespace).API pod logs are shown up until the health check fails:
For completeness sake, I'll add the Executor pod logs since it is restarting repeatedly too.
My Terrakube Helm Chart and App versions:
Let me pause here for now. I can get into my: IAM, VPC, EKS, S3, Cognito, Route 53, and other configurations as needed.
Any help is appreciated. Thanks.