Pods keep on restarting

[rdehuyss]

Hi @tiarebalbi ,

I had some time so I though to quickly look into this. But, for some reason, the pods keep on restarting with me...

rdehuyss@Ronalds-MBP jobrunr-issue-demo % kubectl get pods --namespace=jobrunr            
NAME                                  READY   STATUS    RESTARTS      AGE
demo-deployment-v1-5c5bd8b4c4-6nqb2   0/1     Running   4 (55s ago)   6m55s
demo-deployment-v1-5c5bd8b4c4-7qm98   0/1     Running   4 (45s ago)   6m55s
demo-deployment-v1-5c5bd8b4c4-pkrrn   0/1     Running   4 (60s ago)   7m10s
mongodb-d77b467f8-bgk6d               1/1     Running   0             7m10s

rdehuyss@Ronalds-MBP jobrunr-issue-demo % kubectl logs -f -n jobrunr demo-deployment-v1-5c5bd8b4c4-pkrrn
Setting Active Processor Count to 5
Calculating JVM memory based on 3792468K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx3390124K -XX:MaxMetaspaceSize=95143K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 3792468K, Thread Count: 50, Loaded Class Count: 14384, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=5 -XX:MaxDirectMemorySize=10M -Xmx3390124K -XX:MaxMetaspaceSize=95143K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.0.1)

2023-01-15T17:31:04.048Z  INFO 1 --- [           main] c.t.jobrunr.demo.DemoApplicationKt       : Starting DemoApplicationKt using Java 17.0.5 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2023-01-15T17:31:04.085Z  INFO 1 --- [           main] c.t.jobrunr.demo.DemoApplicationKt       : The following 1 profile is active: "prod"
2023-01-15T17:31:42.776Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data MongoDB repositories in DEFAULT mode.
2023-01-15T17:31:44.282Z  INFO 1 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1420 ms. Found 1 MongoDB repository interfaces.

Any idea what's going on? I'm on a Mac M1.

[rdehuyss]

The minikube logs show something interesting:

==> kube-controller-manager [422fe1a2fd8a] <==
E0115 17:30:41.376595       1 horizontal.go:226] failed to compute desired number of replicas based on listed metrics for Deployment/jobrunr/demo-deployment-v1: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

[tiarebalbi]

Hi @rdehuyss sorry about the delay to get back to you, I'm on PTO but I'm returning later today. I'm also running on M1. I had a quick look at the logs as I left this demo running in a cluster here and I didn't see the error. I'm a little bit concerned that the error might on be happening when using the Mongo DB in cluster mode.

I'm going to deploy a second version using a different mongo setup.