search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 240 forks source link

[BUG] : crc.qcow2 file grows rapidly and make the disk full, throws the error "Cannot get machine state: unexpected libvirt status 3" #3612

Closed vjpiyush123 closed 1 year ago

vjpiyush123 commented 1 year ago

General information

CRC version

# Put `crc version` output here
CRC version 4.12.5

CRC status

# Put `crc status --log-level debug` output here
crc status --log-level debug
DEBU CRC version: 2.15.0+cc05160
DEBU OpenShift version: 4.12.5
DEBU Podman version: 4.3.1
DEBU Running 'crc status'

Cannot get machine state: unexpected libvirt status 3  -- Get this error 

because crc.qcow2 file is growing rapidly and filling up the disk. without use of cluster. 
-rw-r--r--. 1 qemu     qemu     58330185728 Apr 24 12:31 crc.qcow2

ls -rtl <userdir>/.crc/machines/crc/crc.qcow2
-rw-r--r--. 1 qemu qemu 58330185728 Apr 24 12:31 <userdir>/.crc/machines/crc/crc.qcow2
 ls -rtl <userdir>/.crc/cache/crc_libvirt_4.12.5_amd64/crc.qcow2
-rwxr-xr-x. 1 qemu qemu 12694126592 Apr 10 09:38<userdir>/.crc/cache/crc_libvirt_4.12.5_amd64/crc.qcow2

CRC config

# Put `crc config view` output here

 crc config view
- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 90
- memory                                : 30720

 du -h crc.qcow2
55G     crc.qcow2
$ qemu-img info crc.qcow2
image: crc.qcow2
file format: raw
virtual size: 54.3 GiB (58330185728 bytes)
disk size: 54.3 GiB

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux

NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows

Steps to reproduce

  1. Install crc , deploy application
  2. crc.qcow2 image in ~/.crc/machine/crc is keep growing rapidly and fill up disk to 100% , given 70 GB. that also filled up in 4 days of running the crc.

Expected

Actual

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

We have tried it has fixed the issue but it reappears in 4-5days.  Gets the below error 
DEBU Failed to load bundle crc_libvirt_4.12.5_amd64: symlink <userdir>/.crc/cache/crc_libvirt_4.12.5_amd64/oc <userdir>/.crc/bin/oc/oc: no space left on device

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

crc start --log-level debug DEBU CRC version: 2.15.0+cc05160 DEBU OpenShift version: 4.12.5 DEBU Podman version: 4.3.1 DEBU Running 'crc start' DEBU Total memory of system is 33479520256 bytes WARN A new version (2.17.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.17.0/crc-linux-amd64.tar.xz DEBU Checking file: /.crc/machines/crc/.crc-exist DEBU Found binary path at /.crc/bin/crc-driver-libvirt DEBU Launching plugin server for driver libvirt DEBU Plugin server listening at address 127.0.0.1:19667 DEBU () Calling .GetVersion DEBU Using API Version 1 DEBU () Calling .SetConfigRaw DEBU () Calling .GetMachineName DEBU (crc) Calling .GetBundleName DEBU (crc) Calling .GetState DEBU (crc) DBG | time="2023-04-24T13:15:20Z" level=debug msg="Getting current state..." DEBU (crc) DBG | time="2023-04-24T13:15:20Z" level=debug msg="Fetching VM..." DEBU Making call to close driver server DEBU (crc) Calling .Close DEBU Successfully made call to close driver server DEBU Making call to close connection to plugin binary DEBU (crc) DBG | time="2023-04-24T13:15:20Z" level=debug msg="Closing plugin on server side" DEBU Checking if systemd-resolved.service is running DEBU Running 'systemctl status systemd-resolved.service' DEBU Command failed: exit status 3 DEBU stdout: systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; vendor preset: disabled) Active: inactive (dead) Docs: man:systemd-resolved.service(8) https://www.freedesktop.org/wiki/Software/systemd/resolved https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients DEBU stderr: INFO Checking if running as non-root INFO Checking if running inside WSL2 INFO Checking if crc-admin-helper executable is cached DEBU Running '/.crc/bin/crc-admin-helper-linux --version' DEBU Found crc-admin-helper-linux version 0.0.12 DEBU crc-admin-helper executable already cached INFO Checking if running on a supported CPU architecture DEBU GOARCH is amd64 GOOS is linux INFO Checking minimum RAM requirements DEBU Total memory of system is 33479520256 bytes INFO Checking if crc executable symlink exists INFO Checking if Virtualization is enabled DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo DEBU CPU virtualization flags are good INFO Checking if KVM is enabled DEBU Checking if /dev/kvm exists DEBU /dev/kvm was found INFO Checking if libvirt is installed DEBU Checking if 'virsh' is available DEBU 'virsh' was found in /usr/bin/virsh DEBU Checking 'virsh capabilities' for libvirtd/qemu availability DEBU Running 'virsh --readonly --connect qemu:///system capabilities' DEBU Found x86_64 hypervisor with 'hvm' capabilities INFO Checking if user is part of libvirt group DEBU Checking if current user is part of the libvirt group DEBU Current user is already in the libvirt group INFO Checking if active user/process is currently part of the libvirt group INFO Checking if libvirt daemon is running DEBU Checking if libvirtd service is running DEBU Running 'systemctl status virtqemud.socket' DEBU Command failed: exit status 3 DEBU stdout: virtqemud.socket - Libvirt qemu local socket Loaded: loaded (/usr/lib/systemd/system/virtqemud.socket; disabled; vendor preset: disabled) Active: inactive (dead) Listen: /run/libvirt/virtqemud-sock (Stream) DEBU stderr: DEBU virtqemud.socket is neither running nor listening DEBU Running 'systemctl status libvirtd.socket' DEBU libvirtd.socket is running INFO Checking if a supported libvirt version is installed DEBU Checking if libvirt version is >=3.4.0 DEBU Running 'virsh -v' INFO Checking if crc-driver-libvirt is installed DEBU Checking if crc-driver-libvirt is installed DEBU Running '/.crc/bin/crc-driver-libvirt version' DEBU Found crc-driver-libvirt version 0.13.5 DEBU crc-driver-libvirt is already installed INFO Checking crc daemon systemd socket units DEBU Checking crc daemon systemd socket units DEBU Checking if crc-http.socket is running DEBU Running 'systemctl --user status crc-http.socket' DEBU crc-http.socket is running DEBU Checking if crc-http.socket has the expected content DEBU Checking if crc-vsock.socket is running DEBU Running 'systemctl --user status crc-vsock.socket' DEBU crc-vsock.socket is running DEBU Checking if crc-vsock.socket has the expected content INFO Checking if systemd-networkd is running DEBU Checking if systemd-networkd.service is running DEBU Running 'systemctl status systemd-networkd.service' DEBU Command failed: exit status 4 DEBU stdout: DEBU stderr: Unit systemd-networkd.service could not be found. DEBU systemd-networkd.service is not running INFO Checking if NetworkManager is installed DEBU Checking if 'nmcli' is available DEBU 'nmcli' was found in /usr/bin/nmcli INFO Checking if NetworkManager service is running DEBU Checking if NetworkManager.service is running DEBU Running 'systemctl status NetworkManager.service' DEBU NetworkManager.service is already running INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists DEBU Checking NetworkManager configuration DEBU NetworkManager configuration is good INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists DEBU Checking dnsmasq configuration DEBU dnsmasq configuration is good INFO Checking if libvirt 'crc' network is available DEBU Checking if libvirt 'crc' network exists DEBU Running 'virsh --connect qemu:///system net-info crc' DEBU Checking if libvirt 'crc' definition is up to date DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc' DEBU libvirt 'crc' network has the expected value INFO Checking if libvirt 'crc' network is active DEBU Checking if libvirt 'crc' network is active DEBU Running 'virsh --connect qemu:///system net-info crc' DEBU libvirt 'crc' network is already active DEBU Checking file: /.crc/machines/crc/.crc-exist DEBU Failed to load bundle crc_libvirt_4.12.5_amd64: symlink /.crc/cache/crc_libvirt_4.12.5_amd64/oc /.crc/bin/oc/oc: no space left on device INFO Downloading bundle: crc_libvirt_4.12.5_amd64... DEBU Downloading https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/4.12.5/crc_libvirt_4.12.5_amd64.crcbundle to /.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle

praveenkumar commented 1 year ago

@vjpiyush123 do you have some workload deployed on the crc which might be filling the disk?

vjpiyush123 commented 1 year ago

@praveenkumar - We have deployed but it's ideal, no operation is performed even though disk usage keeps growing. The core is disabled so no core files are generated by the application. what we have observed is overlay size is keep growing. Do we have any option to do a cleanup that can remove any unwanted stuff?

cfergeau commented 1 year ago

If you want to understand better what is using disk space, you could ssh into the instance https://github.com/crc-org/crc/wiki/Debugging-guide#entering-the-vm and investigate with du -sh *. Could be the logs which are growing?

One way of limiting disk usage is to not set disk-size 90 as shown in your config.

praveenkumar commented 1 year ago

@vjpiyush123 if you are still facing this issue can you try latest version and create a new issue?