search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.27k stars 242 forks source link

Failed to start CRC #2220

Closed abkieling closed 3 years ago

abkieling commented 3 years ago

General information

CRC version

CodeReady Containers version: 1.24.0+5f06e84b
OpenShift version: 4.7.2 (embedded in executable)

CRC status

DEBU CodeReady Containers version: 1.24.0+5f06e84b 
DEBU OpenShift version: 4.7.2 (embedded in executable) 
DEBU Running 'crc status'                         
DEBU Checking file: /home/akieling/.crc/machines/crc/.crc-exist 
DEBU Checking file: /home/akieling/.crc/machines/crc/.crc-exist 
DEBU Found binary path at /home/akieling/.crc/bin/crc-driver-libvirt 
DEBU Launching plugin server for driver libvirt   
DEBU Plugin server listening at address 127.0.0.1:37509 
DEBU () Calling .GetVersion                       
DEBU Using API Version 1                          
DEBU () Calling .SetConfigRaw                     
DEBU () Calling .GetMachineName                   
DEBU (crc) Calling .GetState                      
DEBU (crc) DBG | time="2021-04-13T16:48:12-03:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2021-04-13T16:48:12-03:00" level=debug msg="Fetching VM..." 
DEBU (crc) Calling .GetBundleName                 
DEBU (crc) Calling .GetIP                         
DEBU (crc) DBG | time="2021-04-13T16:48:12-03:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2021-04-13T16:48:12-03:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2021-04-13T16:48:12-03:00" level=debug msg="IP address: 192.168.130.11" 
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 
DEBU Using ssh private keys: [/home/akieling/.crc/machines/crc/id_ecdsa /home/akieling/.crc/cache/crc_libvirt_4.7.2/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output: 32737570816 9833512960 /sysroot 
DEBU Running SSH command: <hidden>                
DEBU SSH command failed                           
DEBU cannot get OpenShift status: ssh command error:
command : timeout 5s oc get co -ojson --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1\n 
DEBU Making call to close driver server           
DEBU (crc) Calling .Close                         
DEBU (crc) DBG | time="2021-04-13T16:48:13-03:00" level=debug msg="Closing plugin on server side" 
DEBU Successfully made call to close driver server 
DEBU Making call to close connection to plugin binary 
CRC VM:          Running
OpenShift:       Unreachable (v4.7.2)
Disk Usage:      9.834GB of 32.74GB (Inside the CRC VM)
Cache Usage:     13.17GB
Cache Directory: /home/akieling/.crc/cache

CRC config

- consent-telemetry                     : yes

Host Operating System

NAME=Fedora
VERSION="33 (Workstation Edition)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

  1. Run 'crc start'

Expected

Successful start of the VM.

Actual

Failed with the following output:

INFO Starting CodeReady Containers VM for OpenShift 4.7.2... 
Failed to connect to the CRC VM with SSH -- host might be unreachable: Temporary error: ssh command error:
command : exit 0
err     : dial tcp 192.168.130.11:22: connect: no route to host\n (x98)

Logs

https://gist.github.com/abkieling/9771714dce0e2739c7e6c6a1331f44d0

praveenkumar commented 3 years ago

@abkieling Looking at the logs looks which you shared as part of gist having issue with kubelet service which is failed due to some reason. Are you able to get same issue/error even you perform crc delete => crc start consistently? 

DEBU Making call to close connection to plugin binary 
Error starting kubelet: Executing systemctl action failed:  ssh command error:
command : sudo systemctl start kubelet
err     : Process exited with status 1\n: Job for kubelet.service failed because a timeout was exceeded.
See "systemctl status kubelet.service" and "journalctl -xe" for details.
abkieling commented 3 years ago

Yes, the same error happens consistently.

praveenkumar commented 3 years ago

@abkieling Can you try https://github.com/code-ready/crc/wiki/Debugging-guide to login to the VM and provide us sudo journalctl -u kubelet so we have an idea why it is failing for you?

abkieling commented 3 years ago

I've started crc today and it didn't fail. See the console output below.

WARN A new version (1.25.0) has been published on https://cloud.redhat.com/openshift/create/local 
INFO Checking if running as non-root              
INFO Checking if podman remote executable is cached 
INFO Checking if admin-helper executable is cached 
INFO Checking minimum RAM requirements            
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Starting CodeReady Containers VM for OpenShift 4.7.2... 
INFO CodeReady Containers VM is running           
INFO Generating new SSH Key pair ...              
INFO Updating authorized keys ...                 
INFO Copying kubeconfig file to instance dir ...  
INFO Starting network time synchronization in CodeReady Containers VM 
INFO Check internal and public DNS query ...      
INFO Check DNS query from host ...                
INFO Adding user's pull secret to instance disk... 
INFO Verifying validity of the kubelet certificates ... 
INFO Starting OpenShift kubelet service           
INFO Kubelet client certificate has expired, renewing it... [will take up to 8 minutes] 
INFO Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 8 minutes] 
INFO Waiting for kube-apiserver availability... [takes around 2min] 
INFO Adding user's pull secret to the cluster ... 
INFO Updating cluster ID ...                      
INFO Starting OpenShift cluster ... [waiting for the cluster to stabilize] 
INFO 4 operators are progressing: authentication, image-registry, openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 3 operators are progressing: image-registry, kube-apiserver, openshift-controller-manager 
INFO 2 operators are progressing: kube-apiserver, openshift-controller-manager 
INFO Operator kube-apiserver is progressing       
INFO Operator kube-apiserver is progressing       
INFO Operator kube-apiserver is progressing       
INFO Operator kube-apiserver is progressing       
INFO Operator kube-apiserver is progressing       
INFO All operators are available. Ensuring stability ... 
INFO Operators are stable (2/3) ...               
INFO Operators are stable (3/3) ...               
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: wo7m6-BUrLf-yVfEV-FapPA

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
abkieling commented 3 years ago

The issue is not happening anymore. Closing it.

abkieling commented 3 years ago

Reopening issue due to error happening again:

~ crc start
INFO Checking if running as non-root              
INFO Checking if admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
CodeReady Containers requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://cloud.redhat.com/openshift/create/local.
? Please enter the pull secret **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
INFO Loading bundle: crc_libvirt_4.7.8.crcbundle ... 
INFO Creating CodeReady Containers VM for OpenShift 4.7.8... 
INFO CodeReady Containers instance is running with IP  
Failed to connect to the CRC VM with SSH -- host might be unreachable: Temporary error: ssh command error:
command : exit 0
err     : dial tcp :22: connect: connection refused\n (x293)
abkieling commented 3 years ago

Here is screenshot of the VM's console: Screenshot_crc_2021-05-06_08:05:32

abkieling commented 3 years ago

I wasn't able to login with the following command:

~ ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11
Warning: Identity file /home/akieling/.crc/machines/crc/id_ecdsa not accessible: No such file or directory.
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
core@192.168.130.11: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
cfergeau commented 3 years ago

At this point, the ssh key is in ~/.crc/cache/crc_libvirt_4.7.xx. Can you check journalctl on the host? For mentions of unstable clock/tsc in relation with kvm

abkieling commented 3 years ago

OK, I managed to login to the VM and here is the output of sudo journalctl -u kubelet:

~ ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 47.83.202104161442-0
  Part of OpenShift 4.7, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.7/architecture/architecture-rhcos.html

---
[systemd]
Failed Units: 1
  systemd-hostnamed.service
[core@crc-gm7cm-master-0 ~]$ sudo journalctl -u kubelet
-- Logs begin at Fri 2021-04-30 08:04:06 UTC, end at Thu 2021-05-06 12:03:39 UTC. --
May 06 11:58:09 crc-gm7cm-master-0 systemd[1]: Starting Kubernetes Kubelet...
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --con>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --add-dir-header="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --address="0.0.0.0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --allowed-unsafe-sysctls="[]"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --alsologtostderr="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --anonymous-auth="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --application-metrics-count-limit="100"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --authentication-token-webhook="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --authorization-mode="AlwaysAllow"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --azure-container-registry-config=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cgroup-driver="cgroupfs"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cgroup-root=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cgroups-per-qos="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --chaos-chance="0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --client-ca-file=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cloud-config=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cloud-provider=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cluster-dns="[]"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cluster-domain=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cni-bin-dir="/opt/cni/bin"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cni-cache-dir="/var/lib/cni/cache"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cni-conf-dir="/etc/cni/net.d"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --container-log-max-files="5"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --container-log-max-size="10Mi"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --container-runtime="remote"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --containerd="/run/containerd/containerd.sock"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --containerd-namespace="k8s.io"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --contention-profiling="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cpu-cfs-quota="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cpu-cfs-quota-period="100ms"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cpu-manager-policy="none"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --cpu-manager-reconcile-period="10s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker="unix:///var/run/docker.sock"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-env-metadata-whitelist=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-only="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-root="/var/lib/docker"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-tls="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-tls-ca="ca.pem"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-tls-cert="cert.pem"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --docker-tls-key="key.pem"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --dynamic-config-dir=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enable-cadvisor-json-endpoints="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enable-controller-attach-detach="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enable-debugging-handlers="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enable-load-reader="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enable-server="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --enforce-node-allocatable="[pods]"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --event-burst="10"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --event-qps="5"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --event-storage-age-limit="default=0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --event-storage-event-limit="default=0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-max-pod-grace-period="0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-minimum-reclaim=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-pressure-transition-period="5m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-soft=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --eviction-soft-grace-period=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --exit-on-lock-contention="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-allocatable-ignore-eviction="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-check-node-capabilities-before-mount="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-kernel-memcg-notification="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-logging-sanitization="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --experimental-mounter-path=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --fail-swap-on="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --feature-gates=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --file-check-frequency="20s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --global-housekeeping-interval="1m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --hairpin-mode="promiscuous-bridge"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --healthz-bind-address="127.0.0.1"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --healthz-port="10248"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --help="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --hostname-override=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --housekeeping-interval="10s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --http-check-frequency="20s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-credential-provider-bin-dir=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-credential-provider-config=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-gc-high-threshold="85"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-gc-low-threshold="80"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-pull-progress-deadline="1m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --image-service-endpoint=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --iptables-drop-bit="15"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --iptables-masquerade-bit="14"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --keep-terminated-pod-volumes="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kernel-memcg-notification="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kube-api-burst="10"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kube-api-qps="5"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kube-reserved=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kube-reserved-cgroup=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --kubelet-cgroups=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --lock-file=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-backtrace-at=":0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-cadvisor-usage="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-dir=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-file=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-file-max-size="1800"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --log-flush-frequency="5s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --logging-format="text"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --logtostderr="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --make-iptables-util-chains="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --manifest-url=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --manifest-url-header=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --master-service-namespace="default"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --max-open-files="1000000"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --max-pods="110"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --maximum-dead-containers="-1"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --maximum-dead-containers-per-container="1"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --minimum-container-ttl-duration="6m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --minimum-image-ttl-duration="2m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --network-plugin=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --network-plugin-mtu="0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --node-ip="192.168.126.11"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --node-labels="node-role.kubernetes.io/master=,node.openshift.io/os_id=rhcos"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --node-status-max-images="50"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --node-status-update-frequency="10s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --non-masquerade-cidr="10.0.0.0/8"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --one-output="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --oom-score-adj="-999"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --pod-cidr=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sh>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --pod-manifest-path=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --pod-max-pids="-1"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --pods-per-core="0"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --port="10250"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --protect-kernel-defaults="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --provider-id=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --qos-reserved=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --read-only-port="10255"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --really-crash-for-testing="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --redirect-container-streaming="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --register-node="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --register-schedulable="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --register-with-taints="node-role.kubernetes.io/master=:NoSchedule"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --registry-burst="10"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --registry-burst="10"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --registry-qps="5"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --reserved-cpus=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --resolv-conf="/etc/resolv.conf"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --root-dir="/var/lib/kubelet"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --rotate-certificates="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --rotate-server-certificates="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --runonce="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --runtime-request-timeout="2m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --serialize-image-pulls="true"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --skip-headers="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --skip-log-headers="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --stderrthreshold="2"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-buffer-duration="1m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-db="cadvisor"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-host="localhost:8086"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-password="root"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-secure="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-table="stats"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --storage-driver-user="root"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --sync-frequency="1m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --system-cgroups=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --system-reserved=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --system-reserved-cgroup=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --tls-cert-file=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --tls-cipher-suites="[]"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --tls-min-version=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --tls-private-key-file=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --topology-manager-policy="none"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --topology-manager-scope="container"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --v="2"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --version="false"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --vmodule=""
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 flags.go:59] FLAG: --volume-stats-agg-period="1m0s"
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 feature_gate.go:244] feature gates: &{map[]}
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 server.go:191] Warning: For remote container runtime, --pod-infra-container-image is ignored in k>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --con>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SCTPSupport=true. It will be removed in a future rel>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SupportPodPidsLimit=true. It will be removed in a fu>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRoleBehavior:fals>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SCTPSupport=true. It will be removed in a future rel>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SupportPodPidsLimit=true. It will be removed in a fu>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRoleBehavior:fals>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 mount_linux.go:202] Detected OS with systemd
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 server.go:416] Version: v1.20.0+7d0a2b2
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SCTPSupport=true. It will be removed in a future rel>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SupportPodPidsLimit=true. It will be removed in a fu>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRoleBehavior:fals>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SCTPSupport=true. It will be removed in a future rel>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: W0506 11:58:09.959895    2730 feature_gate.go:236] Setting GA feature gate SupportPodPidsLimit=true. It will be removed in a fu>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 feature_gate.go:244] feature gates: &{map[APIPriorityAndFairness:true LegacyNodeRoleBehavior:fals>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 server.go:837] Client rotation is on, will bootstrap in background
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 bootstrap.go:84] Current kubeconfig file contents are still valid, no bootstrap necessary
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 server.go:881] Starting client certificate rotation.
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 certificate_manager.go:282] Certificate rotation is enabled.
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 certificate_manager.go:556] Certificate expiration is 2021-05-30 07:40:25 +0000 UTC, rotation dea>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 certificate_manager.go:288] Waiting 424h0m35.943847156s for next certificate rotation
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/ku>
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/kubelet-ca.crt
May 06 11:58:09 crc-gm7cm-master-0 hyperkube[2730]: I0506 11:58:09.959895    2730 manager.go:165] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.s>
May 06 12:03:36 crc-gm7cm-master-0 systemd[1]: kubelet.service: start operation timed out. Terminating.
May 06 12:03:36 crc-gm7cm-master-0 hyperkube[2730]: I0506 12:03:36.356895    2730 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/etc/kubernetes/kubelet-ca.crt
abkieling commented 3 years ago

And today crc started without errors. The issue doesn't happen all the time.

cfergeau commented 3 years ago

Still curious about this if this happens again

Can you check journalctl on the host? For mentions of unstable clock/tsc in relation with kvm

VivekBuzruk commented 3 years ago

Hi!

I also get this same error (today frequently). Find output of "sudo journalctl -u kubelet:" from Virtual machine in the attached file.

journalctl.log

CRC installed in CentOS Linux release 7.9.2009