[BUG] The node was low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 4398100Ki.

molivo123 commented 5 months ago

General information

OS: Windows
Hypervisor: Hyper-V
Did you run crc setup before starting it (Yes/No)? Yes
Running CRC on: Laptop

CRC version

CRC version: 2.37.1+36d451
OpenShift version: 4.15.14

CRC status

CRC VM:          Running
OpenShift:       Running (v4.15.14)
RAM Usage:       8.528GB of 15.61GB
Disk Usage:      27.62GB of 32.68GB (Inside the CRC VM)
Cache Usage:     29.31GB
Cache Directory: C:\Users\molivo\.crc\cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- memory                                : 15258

Host Operating System

Host Name:                 PRECISION-1016
OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.19045 N/A Build 19045
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:          admin
Registered Organization:
Product ID:                00355-61092-94842-AAOEM
Original Install Date:     6/5/2024, 11:42:29 AM
System Boot Time:          6/12/2024, 8:51:26 AM
System Manufacturer:       Dell Inc.
System Model:              Precision 7680
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 183 Stepping 1 GenuineIntel ~2100 Mhz
BIOS Version:              Dell Inc. 1.11.0, 3/6/2024
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-08:00) Pacific Time (US & Canada)
Total Physical Memory:     65,221 MB
Available Physical Memory: 34,967 MB
Virtual Memory: Max Size:  74,949 MB
Virtual Memory: Available: 43,513 MB
Virtual Memory: In Use:    31,436 MB
Page File Location(s):     C:\pagefile.sys
Logon Server:              \\DC2

Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Steps to reproduce

setup/start crc

apply .yaml file with the following constraints to increase storage:

apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources
spec:
limits:
- type: "Container"
  max:
    pods: "6"
    cpu: "8"
    memory: "16Gi"
    ephemeral-storage: "14Gi"
  min:
    cpu: "8"
    memory: "10Gi"
    ephemeral-storage: "18Gi"
    pods: "4"

create pipeline in Openshift console with PVC storage
Pipeline runs for about 5 mins with no log, just the above error in the events

Expected

Pipeline to run

Actual

low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 4398100Ki.

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

anjannath commented 5 months ago

@molivo123 can you try to increase the disk size using crc config set disk-size 50 by default we allocate 31gb of disk space for the crc vm, of which only ~5gb is available on a fresh install, if you need more space you have to increase it using the config option

molivo123 commented 5 months ago

@anjannath will try this and let you know, thank you for the reply

molivo123 commented 5 months ago

That seemed to help fix my initial issue, however I am now facing an issue with the git-clone clustertask having permission errors

+ test -z ''
+ git config --global --add safe.directory /workspace/output
+ /ko-app/git-init -url=https://github.com/tektoncd/website -revision= -refspec= -path=/workspace/output/ -sslVerify=true -submodules=true -depth=1 -sparseCheckoutDirectories=
{"level":"error","ts":1718296905.603852,"caller":"git/git.go:53","msg":"Error running git [init /workspace/output/]: exit status 1\n/workspace/output/.git: Permission denied\n","stacktrace":"github.com/tektoncd-catalog/git-clone/git-init/git.run\n\t/go/src/github.com/tektoncd-catalog/git-clone/image/git-init/git/git.go:53\ngithub.com/tektoncd-catalog/git-clone/git-init/git.Fetch\n\t/go/src/github.com/tektoncd-catalog/git-clone/image/git-init/git/git.go:88\nmain.main\n\t/go/src/github.com/tektoncd-catalog/git-clone/image/git-init/main.go:52\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250"}
{"level":"fatal","ts":1718296905.60389,"caller":"git-init/main.go:53","msg":"Error fetching git repository: exit status 1","stacktrace":"main.main\n\t/go/src/github.com/tektoncd-catalog/git-clone/image/git-init/main.go:53\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250"}

Seems to be mentioned here: https://issues.redhat.com/browse/SRVKP-3301

patriot1burke commented 4 months ago

Any reason why the default value isn't higher? How do you expect anybody to do anything meaningful with CRC out of the box? I ran into this issue on the 2nd build of a Hello World quarkus application. Nothing else running.

patriot1burke commented 4 months ago

So, what I don't get is why would the 1st build succeed, but not the second? Is there some leak of ephemeral storage going on?

crc-org / crc