search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 242 forks source link

[BUG] - Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x31) #4156

Closed aolloh closed 5 months ago

aolloh commented 6 months ago

General information

CRC version

$ crc version
CRC version: 2.35.0+3956e8
OpenShift version: 4.15.10
Podman version: 4.4.4

CRC status

$ crc status --log-level debug
DEBU CRC version: 2.35.0+3956e8                   
DEBU OpenShift version: 4.15.10                   
DEBU Podman version: 4.4.4                        
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Unreachable (v4.15.10)
RAM Usage:       5.205GB of 12.54GB
Disk Usage:      19.81GB of 32.68GB (Inside the CRC VM)
Cache Usage:     27.27GB
Cache Directory: /home/student/.crc/cache

CRC config

$ crc config view
- consent-telemetry                     : yes

Host Operating System

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

CRC Setup

[student@openshift crc-linux-2.35.0-amd64]$ crc setup
CRC is constantly improving and we would like to know more about usage (more details at https://developers.redhat.com/article/tool-data-collection)
Your preference can be changed manually if desired using 'crc config set consent-telemetry <yes/no>'
Would you like to contribute anonymous usage statistics? [y/N]: y
Thanks for helping us! You can disable telemetry with the command 'crc config set consent-telemetry no'.
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.15.10_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Caching crc-admin-helper executable          
INFO Using root access: Changing ownership of /home/student/.crc/bin/crc-admin-helper-linux 
INFO Using root access: Setting suid for /home/student/.crc/bin/crc-admin-helper-linux 
INFO Checking if running on a supported CPU architecture 
INFO Checking if crc executable symlink exists    
INFO Creating symlink for crc executable          
INFO Checking minimum RAM requirements            
INFO Check if Podman binary exists in: /home/student/.crc/bin/oc 
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Installing libvirt service and dependencies  
INFO Using root access: Installing virtualization packages 
INFO Checking if user is part of libvirt group    
INFO Adding user to libvirt group                 
INFO Using root access: Adding user to the libvirt group 
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
WARN No active (running) libvirtd systemd unit could be found - make sure one of libvirt systemd units is enabled so that it's autostarted at boot time. 
INFO Starting libvirt service                     
INFO Using root access: Executing systemctl daemon-reload command 
INFO Using root access: Executing systemctl start libvirtd 
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Installing crc-driver-libvirt                
INFO Checking crc daemon systemd service          
INFO Setting up crc daemon systemd service        
INFO Checking crc daemon systemd socket units     
INFO Setting up crc daemon systemd socket units   
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists 
INFO Writing Network Manager config for crc       
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf 
INFO Using root access: Changing permissions for /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf to 644  
INFO Using root access: Executing systemctl daemon-reload command 
INFO Using root access: Executing systemctl reload NetworkManager 
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists 
INFO Writing dnsmasq config for crc               
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/dnsmasq.d/crc.conf 
INFO Using root access: Changing permissions for /etc/NetworkManager/dnsmasq.d/crc.conf to 644  
INFO Using root access: Executing systemctl daemon-reload command 
INFO Using root access: Executing systemctl reload NetworkManager 
INFO Checking if libvirt 'crc' network is available 
INFO Setting up libvirt 'crc' network             
INFO Checking if libvirt 'crc' network is active  
INFO Starting libvirt 'crc' network               
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/student/.crc/cache/crc_libvirt_4.15.10_amd64.crcbundle exists 
INFO Getting bundle for the CRC executable        
INFO Downloading bundle: /home/student/.crc/cache/crc_libvirt_4.15.10_amd64.crcbundle... 
4.81 GiB / 4.81 GiB [----------------------------------------------------------------------------------------------------------------] 100.00% 10.02 MiB/s
INFO Uncompressing /home/student/.crc/cache/crc_libvirt_4.15.10_amd64.crcbundle 
crc.qcow2:  20.41 GiB / 20.41 GiB [--------------------------------------------------------------------------------------------------------------] 100.00%
oc:  149.79 MiB / 149.79 MiB [-------------------------------------------------------------------------------------------------------------------] 100.00%
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

Error

[student@openshift crc-linux-2.35.0-amd64]$ crc start -m 12288 -p ../pull-secret.txt 
INFO Using bundle path /home/student/.crc/cache/crc_libvirt_4.15.10_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking if crc executable symlink exists    
INFO Checking minimum RAM requirements            
INFO Check if Podman binary exists in: /home/student/.crc/bin/oc 
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd socket units     
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists 
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists 
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Loading bundle: crc_libvirt_4.15.10_amd64... 
INFO Creating CRC VM for OpenShift 4.15.10...     
INFO Generating new SSH key pair...               
INFO Generating new password for the kubeadmin user 
INFO Starting CRC VM for openshift 4.15.10...     
INFO CRC instance is running with IP 192.168.130.11 
INFO CRC VM is running                            
INFO Updating authorized keys...                  
INFO Configuring shared directories               
INFO Check internal and public DNS query...       
INFO Check DNS query from host...                 
INFO Verifying validity of the kubelet certificates... 
INFO Starting kubelet service                     
INFO Waiting for kube-apiserver availability... [takes around 2min] 
INFO Adding user's pull secret to the cluster...  
INFO Updating SSH key to machine config resource... 
INFO Waiting until the user's pull secret is written to the instance disk... 
Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x31)
praveenkumar commented 6 months ago

This looks like dup to https://github.com/crc-org/crc/issues/4110 and can you try https://github.com/crc-org/crc/issues/4110#issuecomment-2085562237 workaround?

aolloh commented 6 months ago

@praveenkumar systemd-resolved.service doesn't exist

$ sudo systemctl status systemd-resolved.service
Unit systemd-resolved.service could not be found.

$ sudo systemctl list-unit-files | grep resolved
praveenkumar commented 5 months ago

@aolloh did you try the latest version of crc which is 2.36.x it have some fixes around resolved/networkd side.

aolloh commented 5 months ago

@praveenkumar Yes, I had a same error with the latest version

[student@openshift crc-linux-2.36.0-amd64]$ crc version
CRC version: 2.36.0+27c493
OpenShift version: 4.15.12
Podman version: 4.4.4
praveenkumar commented 5 months ago

@aolloh I didn't notice that you are using nested virtualization, any reason you are not able to use your host system instead having a RHEL VM in virtualbox and then using crc? We don't test the nested virt setup.

    OS: Linux
    Hypervisor: Virtualbox
    Did you run crc setup before starting it (Yes/No)? yes
    Running CRC on: VM
aolloh commented 5 months ago

@praveenkumar I wanted to test it in a seperate environment (VM)

praveenkumar commented 5 months ago

@aolloh As I said we don't have any test in place for nested virt stuff so there might be many reason it fails like IO/CPU/memory resources, and it is hard to debug. I would suggest you to use the host instead of separate VM for crc and see if that works.

aolloh commented 5 months ago

@praveenkumar it works on host. Thanks