etanshaul
commented
4 years ago Did you dig into the pod state? I wonder if this is a disk pressure type issue
Closed matthewmichihara closed 3 years ago
etanshaul
commented
4 years ago Did you dig into the pod state? I wonder if this is a disk pressure type issue
matthewmichihara
commented
4 years ago Some debug info:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/java-guestbook-backend-968b47b4b-6klm7 0/1 Pending 0 2m49s
pod/java-guestbook-backend-968b47b4b-d6f6x 0/1 Evicted 0 3m30s
pod/java-guestbook-frontend-576df857b-jplql 0/1 Pending 0 2m46s
pod/java-guestbook-frontend-576df857b-wgsl7 0/1 Evicted 0 3m30s
pod/java-guestbook-mongodb-7b5c745c8f-f7mvg 0/1 Pending 0 2m37s
pod/java-guestbook-mongodb-7b5c745c8f-wn448 0/1 Evicted 0 3m30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/java-guestbook-backend ClusterIP 10.111.82.205 <none> 8080/TCP 3m30s
service/java-guestbook-frontend LoadBalancer 10.107.32.223 localhost 80:31283/TCP 3m30s
service/java-guestbook-mongodb ClusterIP 10.100.58.38 <none> 27017/TCP 3m30s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 40d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/java-guestbook-backend 0/1 1 0 3m30s
deployment.apps/java-guestbook-frontend 0/1 1 0 3m30s
deployment.apps/java-guestbook-mongodb 0/1 1 0 3m30s
NAME DESIRED CURRENT READY AGE
replicaset.apps/java-guestbook-backend-968b47b4b 1 1 0 3m30s
replicaset.apps/java-guestbook-frontend-576df857b 1 1 0 3m30s
replicaset.apps/java-guestbook-mongodb-7b5c745c8f 1 1 0 3m30s
$ kubectl describe pod/java-guestbook-backend-968b47b4b-d6f6x
Name: java-guestbook-backend-968b47b4b-d6f6x
Namespace: default
Priority: 0
Node: docker-desktop/
Start Time: Mon, 20 Apr 2020 18:22:38 -0400
Labels: app=java-guestbook
app.kubernetes.io/managed-by=skaffold-v1.8.0
pod-template-hash=968b47b4b
skaffold.dev/builder=local
skaffold.dev/cleanup=true
skaffold.dev/deployer=kubectl
skaffold.dev/docker-api-version=1.40
skaffold.dev/run-id=cba5efe9-6a1f-49af-bcd6-b8cb9b30ad35
skaffold.dev/tag-policy=sha256
skaffold.dev/tail=true
tier=backend
Annotations: <none>
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage. Container backend was using 96Ki, which exceeds its request of 0.
IP:
IPs: <none>
Controlled By: ReplicaSet/java-guestbook-backend-968b47b4b
Init Containers:
init-db-ready:
Image: mongo:4
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
Args:
echo "Waiting for mongodb at java-guestbook-mongodb:27017 to go live before the BE...";
until (mongo --host java-guestbook-mongodb:27017 >/dev/null) do echo "Waiting for connection for 2 sec."; sleep 2; done
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q4ngt (ro)
Containers:
backend:
Image: java-guestbook-backend:136913d20043db17c72305fcf66d1f498e17476c96ec3f4323823892542fabcc
Port: 8080/TCP
Host Port: 0/TCP
Environment:
PORT: 8080
GUESTBOOK_DB_ADDR: java-guestbook-mongodb:27017
JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q4ngt (ro)
Volumes:
default-token-q4ngt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-q4ngt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m21s default-scheduler Successfully assigned default/java-guestbook-backend-968b47b4b-d6f6x to docker-desktop
Normal Pulling 2m20s kubelet, docker-desktop Pulling image "mongo:4"
Normal Pulled 115s kubelet, docker-desktop Successfully pulled image "mongo:4"
Normal Created 115s kubelet, docker-desktop Created container init-db-ready
Normal Started 115s kubelet, docker-desktop Started container init-db-ready
Normal Pulled 115s kubelet, docker-desktop Container image "java-guestbook-backend:136913d20043db17c72305fcf66d1f498e17476c96ec3f4323823892542fabcc" already present on machine
Normal Created 115s kubelet, docker-desktop Created container backend
Normal Started 115s kubelet, docker-desktop Started container backend
Warning Evicted 100s kubelet, docker-desktop The node was low on resource: ephemeral-storage. Container backend was using 96Ki, which exceeds its request of 0.
Normal Killing 100s kubelet, docker-desktop Stopping container backendI docker system prune -f'd and tried again:
$ skaffold run
Generating tags...
- java-guestbook-backend -> java-guestbook-backend:latest
- java-guestbook-frontend -> java-guestbook-frontend:latest
Checking cache...
- java-guestbook-backend: Not found. Building
- java-guestbook-frontend: Not found. Building
Found [docker-desktop] context, using local docker daemon.
Building [java-guestbook-backend]...
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.springframework.boot:frontend:jar:1.0
[WARNING] 'parent.relativePath' of POM org.springframework.boot:frontend:1.0 (/Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/frontend/pom.xml) points at com.cloudcode.guestbook:java-guestbook instead of org.springframework.boot:spring-boot-starter-parent, please verify your project structure @ line 12, column 13
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------------< org.springframework.boot:backend >------------------
[INFO] Building Guestbook Backend 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- jib-maven-plugin:2.1.0:_skaffold-fail-if-jib-out-of-date (default-cli) @ backend ---
[INFO]
[INFO] ------------------< org.springframework.boot:backend >------------------
[INFO] Building Guestbook Backend 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ backend ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ backend ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ backend ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/backend/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ backend ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ backend ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ backend ---
[INFO] Building jar: /Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/backend/target/backend-1.0.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.2.RELEASE:repackage (repackage) @ backend ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- jib-maven-plugin:2.1.0:dockerBuild (default-cli) @ backend ---
[INFO]
[INFO] Containerizing application to Docker daemon as java-guestbook-backend...
[WARNING] Base image 'gcr.io/distroless/java:8' does not use a specific image digest - build may not be reproducible
[INFO] Getting manifest for base image gcr.io/distroless/java:8...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] Using base image with digest: sha256:e99eb6cf88ca2df69e99bf853d65f125066730e3e9f7a233bd1b7e3523c144cb
[INFO]
[INFO] Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, cloudcode.guestbook.backend.BackendApplication]
[INFO] Loading to Docker daemon...
[INFO]
[INFO] Built image to Docker daemon as java-guestbook-backend
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.620 s
[INFO] Finished at: 2020-04-20T18:32:10-04:00
[INFO] ------------------------------------------------------------------------
Building [java-guestbook-frontend]...
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.springframework.boot:frontend:jar:1.0
[WARNING] 'parent.relativePath' of POM org.springframework.boot:frontend:1.0 (/Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/frontend/pom.xml) points at com.cloudcode.guestbook:java-guestbook instead of org.springframework.boot:spring-boot-starter-parent, please verify your project structure @ line 12, column 13
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] -----------------< org.springframework.boot:frontend >------------------
[INFO] Building Guestbook Frontend 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- jib-maven-plugin:2.1.0:_skaffold-fail-if-jib-out-of-date (default-cli) @ frontend ---
[INFO]
[INFO] -----------------< org.springframework.boot:frontend >------------------
[INFO] Building Guestbook Frontend 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ frontend ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ frontend ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ frontend ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/frontend/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ frontend ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ frontend ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:3.1.1:jar (default-jar) @ frontend ---
[INFO] Building jar: /Users/michihara/Code/repo/cloud-code-samples/java/java-guestbook/frontend/target/frontend-1.0.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.2.RELEASE:repackage (repackage) @ frontend ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- jib-maven-plugin:2.1.0:dockerBuild (default-cli) @ frontend ---
[INFO]
[INFO] Containerizing application to Docker daemon as java-guestbook-frontend...
[WARNING] Base image 'gcr.io/distroless/java:8' does not use a specific image digest - build may not be reproducible
[INFO] Getting manifest for base image gcr.io/distroless/java:8...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] Using base image with digest: sha256:e99eb6cf88ca2df69e99bf853d65f125066730e3e9f7a233bd1b7e3523c144cb
[INFO]
[INFO] Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, cloudcode.guestbook.frontend.FrontendApplication]
[INFO] Loading to Docker daemon...
[INFO]
[INFO] Built image to Docker daemon as java-guestbook-frontend
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.970 s
[INFO] Finished at: 2020-04-20T18:32:16-04:00
[INFO] ------------------------------------------------------------------------
Tags used in deployment:
- java-guestbook-backend -> java-guestbook-backend:136913d20043db17c72305fcf66d1f498e17476c96ec3f4323823892542fabcc
- java-guestbook-frontend -> java-guestbook-frontend:89676bc80b9eb3713ee3d670312c5f9f3a684bf637c20fbf881f43c1968262a2
local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
- service/java-guestbook-backend created
- service/java-guestbook-frontend created
- service/java-guestbook-mongodb created
- deployment.apps/java-guestbook-backend created
- deployment.apps/java-guestbook-frontend created
- deployment.apps/java-guestbook-mongodb created
Waiting for deployments to stabilize...
- deployment/java-guestbook-backend:
- deployment/java-guestbook-frontend:
- deployment/java-guestbook-mongodb:
- deployment/java-guestbook-backend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/java-guestbook-frontend: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/java-guestbook-mongodb: waiting for rollout to finish: 0 of 1 updated replicas are available...
- deployment/java-guestbook-frontend failed. [1/3 deployment(s) still pending] Error: could not stabilize within 2m0s: context deadline exceeded.
- deployment/java-guestbook-backend failed. [2/3 deployment(s) still pending] Error: could not stabilize within 2m0s: context deadline exceeded.
- deployment/java-guestbook-mongodb failed. Error: could not stabilize within 2m0s: context deadline exceeded.
FATA[0137] 3/3 deployment(s) failedBut, it actually did work despite it saying it failed! Navigating to localhost shows the guestbook and things look healthy:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/java-guestbook-backend-56b87db54-9nz57 1/1 Running 0 4m6s
pod/java-guestbook-frontend-679857d688-6tcqh 1/1 Running 0 4m6s
pod/java-guestbook-mongodb-5586bcb885-7xtc8 1/1 Running 0 4m6s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/java-guestbook-backend ClusterIP 10.96.221.206 <none> 8080/TCP 4m6s
service/java-guestbook-frontend LoadBalancer 10.100.41.200 localhost 80:30642/TCP 4m6s
service/java-guestbook-mongodb ClusterIP 10.111.169.237 <none> 27017/TCP 4m6s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5m58s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/java-guestbook-backend 1/1 1 1 4m6s
deployment.apps/java-guestbook-frontend 1/1 1 1 4m6s
deployment.apps/java-guestbook-mongodb 1/1 1 1 4m6s
NAME DESIRED CURRENT READY AGE
replicaset.apps/java-guestbook-backend-56b87db54 1 1 1 4m6s
replicaset.apps/java-guestbook-frontend-679857d688 1 1 1 4m6s
replicaset.apps/java-guestbook-mongodb-5586bcb885 1 1 1 4m6s@nkubala @balopat this feels like it may be a skaffold issue?
tstromberg
commented
4 years ago This looks like it's a Docker for Desktop cluster, rather than a minikube cluster, but the issue is all the same:
Status: Failed
Reason: Evicted
Message: The node was low on resource: ephemeral-storage. Container backend was using 96Ki, which exceeds its request of 0.It would be interesting to see the output of kubectl describe no -A. I'm curious if it detected DiskPressure in this condition, because if so, Skaffold and/or other tools could use it to detect the issue ahead of time.
MemoryPressure False Fri, 24 Apr 2020 10:09:32 -0700 Fri, 24 Apr 2020 08:48:57 -0700 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 24 Apr 2020 10:09:32 -0700 Fri, 24 Apr 2020 08:48:57 -0700 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 24 Apr 2020 10:09:32 -0700 Fri, 24 Apr 2020 08:48:57 -0700 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 24 Apr 2020 10:09:32 -0700 Fri, 24 Apr 2020 08:49:16 -0700 KubeletReady kubelet is posting ready status
...
Capacity:
cpu: 6
ephemeral-storage: 61255492Ki
Allocatable:
cpu: 6
ephemeral-storage: 61255492KiI would be interested to see if the cluster detected any DiskPressure issues.
matthewmichihara
commented
4 years ago Yes this was a Docker cluster. I don't currently have the failing state anymore but will run this command and save its output if I run into it again.
matthewmichihara
commented
4 years ago I did hit this again trying to run the java-guestbook sample app. Here is the output of some commands:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/java-guestbook-backend-bc5ccfb-tj28q 0/1 Evicted 0 3m7s
pod/java-guestbook-backend-bc5ccfb-zczcg 0/1 Pending 0 2m47s
pod/java-guestbook-frontend-795697749-27zpd 0/1 Evicted 0 3m7s
pod/java-guestbook-frontend-795697749-8jqzq 0/1 Evicted 0 2m57s
pod/java-guestbook-frontend-795697749-b865w 0/1 Pending 0 2m56s
pod/java-guestbook-frontend-795697749-fvcfg 0/1 Evicted 0 2m56s
pod/java-guestbook-frontend-795697749-j6kkj 0/1 Evicted 0 2m57s
pod/java-guestbook-frontend-795697749-pqq2t 0/1 Evicted 0 2m57s
pod/java-guestbook-frontend-795697749-s78wd 0/1 Evicted 0 2m57s
pod/java-guestbook-frontend-795697749-sdnq9 0/1 Evicted 0 2m56s
pod/java-guestbook-frontend-795697749-sg7vl 0/1 Evicted 0 2m57s
pod/java-guestbook-mongodb-64cfd79974-sxs5z 1/1 Running 0 3m7s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/java-guestbook-backend ClusterIP 10.96.91.200 <none> 8080/TCP 3m7s
service/java-guestbook-frontend LoadBalancer 10.107.205.194 <pending> 80:31952/TCP 3m7s
service/java-guestbook-mongodb ClusterIP 10.102.70.78 <none> 27017/TCP 3m7s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/java-guestbook-backend 0/1 1 0 3m7s
deployment.apps/java-guestbook-frontend 0/1 1 0 3m7s
deployment.apps/java-guestbook-mongodb 1/1 1 1 3m7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/java-guestbook-backend-bc5ccfb 1 1 0 3m7s
replicaset.apps/java-guestbook-frontend-795697749 1 1 0 3m7s
replicaset.apps/java-guestbook-mongodb-64cfd79974 1 1 1 3m7s
$ kubectl describe no -A
Name: docker-desktop
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=docker-desktop
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 12 May 2020 08:44:58 -0700
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: docker-desktop
AcquireTime: <unset>
RenewTime: Tue, 09 Jun 2020 15:02:19 -0700
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 09 Jun 2020 15:01:40 -0700 Tue, 12 May 2020 08:44:55 -0700 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure True Tue, 09 Jun 2020 15:01:40 -0700 Tue, 09 Jun 2020 14:59:19 -0700 KubeletHasDiskPressure kubelet has disk pressure
PIDPressure False Tue, 09 Jun 2020 15:01:40 -0700 Tue, 12 May 2020 08:44:55 -0700 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 09 Jun 2020 15:01:40 -0700 Tue, 12 May 2020 08:44:55 -0700 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.65.3
Hostname: docker-desktop
Capacity:
cpu: 4
ephemeral-storage: 16447356Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4035056Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 15157883265
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3932656Ki
pods: 110
System Info:
Machine ID: 2c3e131c-00a4-4e7b-9ae6-457e11df6d31
System UUID: a7d34bbb-0000-0000-afb1-4f95473fa6ed
Boot ID: a9adf759-7581-4db1-ae63-a1ddbc6b23ee
Kernel Version: 4.19.76-linuxkit
OS Image: Docker Desktop
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.8
Kubelet Version: v1.16.6-beta.0
Kube-Proxy Version: v1.16.6-beta.0
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default java-guestbook-mongodb-64cfd79974-sxs5z 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m16s
kube-system coredns-5644d7b6d9-5cbbz 100m (2%) 0 (0%) 70Mi (1%) 170Mi (4%) 28d
kube-system coredns-5644d7b6d9-w7jln 100m (2%) 0 (0%) 70Mi (1%) 170Mi (4%) 28d
kube-system etcd-docker-desktop 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d
kube-system kube-apiserver-docker-desktop 250m (6%) 0 (0%) 0 (0%) 0 (0%) 28d
kube-system kube-controller-manager-docker-desktop 200m (5%) 0 (0%) 0 (0%) 0 (0%) 28d
kube-system kube-proxy-gtf6v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d
kube-system kube-scheduler-docker-desktop 100m (2%) 0 (0%) 0 (0%) 0 (0%) 28d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 750m (18%) 0 (0%)
memory 140Mi (3%) 340Mi (8%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasNoDiskPressure 6m35s (x27 over 3h26m) kubelet, docker-desktop Node docker-desktop status is now: NodeHasNoDiskPressure
Normal NodeHasDiskPressure 3m5s (x7 over 3h18m) kubelet, docker-desktop Node docker-desktop status is now: NodeHasDiskPressure
Warning EvictionThresholdMet 2m53s (x25 over 3h18m) kubelet, docker-desktop Attempting to reclaim ephemeral-storage
$
askmeegs
commented
3 years ago Hello, as noted above the EvictionThresholdMet error is what causes the app to not become ready, specifically -
The node was low on resource: ephemeral-storage. Container backend was using 96Ki, which exceeds its request of 0.None of the Java guestbook deployments have resource quotas (eg. cpu, memory, ephemeral storage) listed.
We do not typically set explicit ephemeral-storage quotas in GKE/Kubernetes samples.
For your Docker Desktop cluster specifically, I'd try adding ephemeral-storage quotas to your local Deployments like this - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage
If that doesn't resolve it, this related issue could be useful, there's a small chance existing logs are using up space and indirectly causing this problem: https://access.redhat.com/solutions/4367311
Closing this issue; feel free to reopen if you continue to see errors.
Describe the bug I tried to deploy the
java-guestbooksample to my Kubernetes cluster, but it failed.To Reproduce