search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 242 forks source link

[BUG] connection refused trying to start openshift instance. #4370

Closed yassine0s closed 2 months ago

yassine0s commented 2 months ago

General information

Host Operating System

cat /etc/os-release:
 PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

I followed the official documentation to install openshift locally in my Ubuntu system. After running crc setup and crc start it get stuck in INFO 5 operators are progressing: authentication, console, image-registry, kube-apiserver, kube-controller-manager

then gives me this error and get stuck in starting phase:

ERRO Cannot update kubeconfig: Head "https://oauth-openshift.apps-crc.testing:443": dial tcp 192.168.130.11:443: connect: connection refused Started the OpenShift cluster.

❯ crc config view
- consent-telemetry                     : yes
- cpus                                  : 8
- memory                                : 16384

Logs

Here is the full debug output while starting:


INFO Check DNS query from host...                 
DEBU api.crc.testing resolved to [192.168.130.11] 
WARN Wildcard DNS resolution for apps-crc.testing does not appear to be working 
INFO Verifying validity of the kubelet certificates... 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-08-20T06:19:22+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-08-20T06:20:43+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.crt -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-08-20T06:21:53+00:00 
INFO Starting kubelet service                     
DEBU Using root access: Executing systemctl daemon-reload command 
DEBU Running SSH command: sudo systemctl daemon-reload 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Executing systemctl start kubelet 
DEBU Running SSH command: sudo systemctl start kubelet 
DEBU SSH command results: err: <nil>, output:     
INFO Waiting for kube-apiserver availability... [takes around 2min] 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 1                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 2                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 3                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 4                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME   STATUS   ROLES                         AGE   VERSION
crc    Ready    control-plane,master,worker   36d   v1.29.7+6abe8a1 
DEBU NAME   STATUS   ROLES                         AGE   VERSION
crc    Ready    control-plane,master,worker   36d   v1.29.7+6abe8a1 
INFO Adding proxy configuration to the cluster... 
DEBU Waiting for availability of resource type 'proxy' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get proxy --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME      AGE
cluster   36d 
DEBU NAME      AGE
cluster   36d                  

DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Waiting for availability of resource type 'configmap' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get configmap --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME                       DATA   AGE
kube-root-ca.crt           1      36d
openshift-service-ca.crt   1      36d 
DEBU NAME                       DATA   AGE
kube-root-ca.crt           1      36d
openshift-service-ca.crt   1      36d 
DEBU Running SSH command: timeout 30s oc delete -n openshift-machine-config-operator configmap machine-config-controller --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU Waiting for availability of resource type 'lease' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get lease --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output:     
DEBU                                              
DEBU Running SSH command: timeout 30s oc delete -A lease --all --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: lease.coordination.k8s.io "crc" deleted
lease.coordination.k8s.io "apiserver-yz2phfmetulqy4pemhbirhdnii" deleted
lease.coordination.k8s.io "kube-controller-manager" deleted
lease.coordination.k8s.io "openshift-apiserver-operator-lock" deleted
lease.coordination.k8s.io "cluster-authentication-operator-lock" deleted
lease.coordination.k8s.io "cluster-machine-approver-leader" deleted
lease.coordination.k8s.io "version" deleted
lease.coordination.k8s.io "config-operator-lock" deleted
lease.coordination.k8s.io "console-operator-lock" deleted
lease.coordination.k8s.io "openshift-controller-manager-operator-lock" deleted
lease.coordination.k8s.io "openshift-master-controllers" deleted
lease.coordination.k8s.io "openshift-cluster-etcd-operator-lock" deleted
lease.coordination.k8s.io "openshift-master-controllers" deleted
lease.coordination.k8s.io "kube-apiserver-operator-lock" deleted
lease.coordination.k8s.io "cert-regeneration-controller-lock" deleted
lease.coordination.k8s.io "kube-controller-manager-operator-lock" deleted
lease.coordination.k8s.io "cert-recovery-controller-lock" deleted
lease.coordination.k8s.io "cluster-policy-controller-lock" deleted
lease.coordination.k8s.io "openshift-cluster-kube-scheduler-operator-lock" deleted
lease.coordination.k8s.io "cert-recovery-controller-lock" deleted
lease.coordination.k8s.io "kube-scheduler" deleted
lease.coordination.k8s.io "openshift-kube-storage-version-migrator-operator-lock" deleted
lease.coordination.k8s.io "control-plane-machine-set-leader" deleted
lease.coordination.k8s.io "machine-api-operator" deleted
lease.coordination.k8s.io "machine-config" deleted
lease.coordination.k8s.io "machine-config-controller" deleted
lease.coordination.k8s.io "marketplace-operator-lock" deleted
lease.coordination.k8s.io "ovnkube-identity" deleted
lease.coordination.k8s.io "network-operator-lock" deleted
lease.coordination.k8s.io "packageserver-controller-lock" deleted
lease.coordination.k8s.io "ovn-kubernetes-master" deleted
lease.coordination.k8s.io "openshift-route-controllers" deleted
lease.coordination.k8s.io "service-ca-operator-lock" deleted
lease.coordination.k8s.io "service-ca-controller-lock" deleted 
DEBU Waiting for availability of resource type 'secret' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get secret --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME                       TYPE                      DATA   AGE
builder-dockercfg-5gn6x    kubernetes.io/dockercfg   1      35d
default-dockercfg-5mnnm    kubernetes.io/dockercfg   1      35d
deployer-dockercfg-psgvx   kubernetes.io/dockercfg   1      35d 
DEBU NAME                       TYPE                      DATA   AGE
builder-dockercfg-5gn6x    kubernetes.io/dockercfg   1      35d
default-dockercfg-5mnnm    kubernetes.io/dockercfg   1      35d
deployer-dockercfg-psgvx   kubernetes.io/dockercfg   1      35d 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Waiting for availability of resource type 'machineconfigs' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get machineconfigs --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
00-worker                                          64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-master-container-runtime                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-master-kubelet                                  64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-worker-container-runtime                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-worker-kubelet                                  64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
97-master-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
97-worker-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
98-master-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
98-worker-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-master-generated-registries                     64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-master-ssh                                                                                 3.2.0             36d
99-node-sizing-for-crc                                                                        3.2.0             36d
99-openshift-machineconfig-master-dummy-networks                                              3.2.0             36d
99-worker-generated-registries                     64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-worker-ssh                                                                                 3.2.0             36d
rendered-master-06eeaf7f394d4b524d36a6dc5c2cf082   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m39s
rendered-master-38261eb31663f01348b935bee173aa55   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m33s
rendered-master-78aad039d0f32b56c5eea646f22a3d4b   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m45s
rendered-master-af3c83ca74c63b4b8a3a3381b9851b1f   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             35d
rendered-worker-1cf5deca3e283766ba3f6516ed2529a8   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m36s
rendered-worker-b45d4d2f8d21449c9543a709a1728281   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             35d 
DEBU NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
00-worker                                          64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-master-container-runtime                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-master-kubelet                                  64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-worker-container-runtime                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
01-worker-kubelet                                  64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
97-master-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
97-worker-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
98-master-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
98-worker-generated-kubelet                        64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-master-generated-registries                     64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-master-ssh                                                                                 3.2.0             36d
99-node-sizing-for-crc                                                                        3.2.0             36d
99-openshift-machineconfig-master-dummy-networks                                              3.2.0             36d
99-worker-generated-registries                     64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             36d
99-worker-ssh                                                                                 3.2.0             36d
rendered-master-06eeaf7f394d4b524d36a6dc5c2cf082   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m39s
rendered-master-38261eb31663f01348b935bee173aa55   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m33s
rendered-master-78aad039d0f32b56c5eea646f22a3d4b   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m45s
rendered-master-af3c83ca74c63b4b8a3a3381b9851b1f   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             35d
rendered-worker-1cf5deca3e283766ba3f6516ed2529a8   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             8m36s
rendered-worker-b45d4d2f8d21449c9543a709a1728281   64434515dba546949bcec20be26f3215a4fc5b53   3.4.0             35d 
DEBU Running SSH command: timeout 30s oc get machineconfigs 99-master-ssh -o jsonpath='{.spec.config.passwd.users[0].sshAuthorizedKeys[0]}' --context admin --cluster crc --kubeconfig /opt/kubeconfig 

DEBU retry loop: attempt 0                        
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Waiting for availability of resource type 'secret' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get secret --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME                       TYPE                      DATA   AGE
builder-dockercfg-5gn6x    kubernetes.io/dockercfg   1      35d
default-dockercfg-5mnnm    kubernetes.io/dockercfg   1      35d
deployer-dockercfg-psgvx   kubernetes.io/dockercfg   1      35d 
DEBU NAME                       TYPE                      DATA   AGE
builder-dockercfg-5gn6x    kubernetes.io/dockercfg   1      35d
default-dockercfg-5mnnm    kubernetes.io/dockercfg   1      35d
deployer-dockercfg-psgvx   kubernetes.io/dockercfg   1      35d 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Waiting for availability of resource type 'clusterversion' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get clusterversion --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.7    True        False         36d     Error while reconciling 4.16.7: an unknown error has occurred: MultipleErrors 
DEBU NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.7    True        False         36d     Error while reconciling 4.16.7: an unknown error has occurred: MultipleErrors 
DEBU Running SSH command: timeout 30s oc get clusterversion version -o jsonpath="{['spec']['clusterID']}" --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: 32fafc19-18e1-405e-8b8c-b7ad25f0f9a8 
DEBU Waiting for availability of resource type 'configmaps' 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get configmaps --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: <nil>, output: NAME                       DATA   AGE
kube-root-ca.crt           1      36d
openshift-service-ca.crt   1      36d 
DEBU NAME                       DATA   AGE
kube-root-ca.crt           1      36d
openshift-service-ca.crt   1      36d 
DEBU Running SSH command: timeout 30s oc get configmaps admin-kubeconfig-client-ca -n openshift-config -o jsonpath="{.data.ca-bundle\.crt}" --context admin --cluster crc --kubeconfig /opt/kubeconfig
praveenkumar commented 2 months ago

@yassine0s can you use https://github.com/crc-org/crc/issues/new?assignees=&labels=kind%2Fbug%2C+status%2Fneed+triage&projects=&template=bug_report.md&title=%5BBUG%5D issue template to create the issue, this is missing lot of point and we need more info about the issue before provide any valid solution.

praveenkumar commented 2 months ago

@yassine0s Can you try following on that machine and let us know if that work or hit same error?

$ crc delete
$ crc cleanup
$ crc config set network-mode user
$ crc setup
$ crc start
yassine0s commented 2 months ago

Thanks it is running now. there is only a weird error but I think it is not related: 

DEBU error: Temporary error: pull secret not updated to disk - sleeping 2s 
DEBU retry loop: attempt 75