Closed elixirliao closed 5 months ago
ingress-http-port
will only work with user mode networking - crc config set network-mode user
Thanks for your suggestion, I tested it but network-mode=user didn't work. And I found crc status became Unreachable after I changed it to user and crc start face many "err : ssh: handshake failed: read tcp 127.0.0.1:58356->127.0.0.1:2222: read: connection reset by peer".
[09:23]root@ncblinuxdev01:~$crc start
INFO Using bundle path /root/.crc/cache/crc_libvirt_4.14.8_amd64.crcbundle
INFO Checking if running as non-root
WARN Skipping above check...
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if vsock is correctly configured
INFO Loading bundle: crc_libvirt_4.14.8_amd64...
INFO Starting CRC VM for openshift 4.14.8...
INFO CRC instance is running with IP 127.0.0.1
Failed to connect to the CRC VM with SSH -- virtual machine might be unreachable: Temporary error: ssh command error:
command : exit 0
err : ssh: handshake failed: read tcp 127.0.0.1:55000->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:34424->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:34436->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:34440->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46340->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46346->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46352->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:37222->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:37238->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40850->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40856->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40870->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:51066->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:51078->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:51086->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:58354->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:58370->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49930->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49944->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49954->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:54170->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:54180->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:54184->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:50554->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:50562->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45882->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45888->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45896->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:50732->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:50738->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:50748->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:57158->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:57170->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:42574->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:42590->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:42600->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:34684->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:34700->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:47634->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:47640->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:47646->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:41486->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:41498->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:41514->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46276->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46286->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:46288->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45306->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45308->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:59848->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:59860->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:59870->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:53152->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:53168->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:53180->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:41242->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:41248->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49864->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49876->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:49884->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:35562->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:35576->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45750->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45756->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:45766->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36904->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36910->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36922->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36326->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36340->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:36344->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:56008->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:56024->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40078->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40094->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:40096->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:59472->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:59484->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:58344->127.0.0.1:2222: read: connection reset by peer
Temporary error: ssh command error: command : exit 0 err : ssh: handshake failed: read tcp 127.0.0.1:58356->127.0.0.1:2222: read: connection reset by peer
[09:35]root@ncblinuxdev01:/tmp/nginx$crc config view
[09:35]root@ncblinuxdev01:/tmp/nginx$curl http://nginx-nginx.apps-crc.testing:8080/ curl: (7) Failed to connect to nginx-nginx.apps-crc.testing port 8080: Connection refused
[09:36]root@ncblinuxdev01:/tmp/nginx$crc status CRC VM: Running OpenShift: Unreachable (v4.14.8) Disk Usage: 0B of 0B (Inside the CRC VM) Cache Usage: 18.57GB Cache Directory: /root/.crc/cache
[09:37]root@ncblinuxdev01:/tmp/nginx$crc config unset network-mode Successfully unset configuration property 'network-mode' [09:50]root@ncblinuxdev01:/tmp/nginx$crc status CRC VM: Running OpenShift: Running (v4.14.8) RAM Usage: 21.56GB of 41.06GB Disk Usage: 58.13GB of 128.2GB (Inside the CRC VM) Cache Usage: 18.57GB Cache Directory: /root/.crc/cache
can you try again after doing crc cleanup
and crc setup
, changing the network mode needs the crc daemon to be restarted
This looks like dup of https://github.com/crc-org/crc/issues/4124 one and @vyasgun is working on it.
Marked as duplicate after conversation with Gunjan
@anjannath This messes with the 'state' of the daemon as you re-init. In other cases (like a crash on Linux and restarted by systemd) causes other kinds of issues. This therefore feels like a problem with timing or configuration (the old config is still active? we never unexpose the forward).
@anjannath This messes with the 'state' of the daemon as you re-init. In other cases (like a crash on Linux and restarted by systemd) causes other kinds of issues.
actually we do ask the user to run cleanup
and setup
when network mode is changed, but i think crc stop
would be sufficient, i suspect the user has not yet hit the #4124 issue, but they will eventually when the network mode is properly changed
This therefore feels like a problem with timing or configuration (the old config is still active? we never unexpose the forward).
if the daemon is crashed or killed, the ports are automatically unexposed, as the process died which was doing the forwarding and we don't have any state saved on disk, the default ports (6443, 443, 80) are always exposed again when the daemon is started
Following your suggestions, I performed crc cleanup and crc setup, and then ran crc start with the --log-level debug option. The startup process took a very long time. During this process, the error message 'ssh: handshake failed: read tcp 127.0.0.1:59734->127.0.0.1:2222: read: connection reset by peer' appeared several times, and the debug message 'Temporary error: pull secret not updated to disk - sleeping 2s' appeared 151 times. At one point, I lost hope, but in the end, the startup process completed successfully. The crc status is now 'running'. Subsequent tests on the ingress-http-port 8080 were also successful. I really appreciate your suggestions. However, I still don't quite understand the real purpose of the Network mode (user or system).
@elixirliao What you want to achieve? As you first comment https://github.com/crc-org/crc/issues/4177#issue-2314849022 I can see you were able to deploy the app successfully and have route associated to it. Now all the routes which are created by openshift are by default exposed to port 80
(non-tls one) which is also expecting in your case and you were able to curl http://ngins-nginx.apps-crc.testing
perform this.
@elixirliao What you want to achieve? As you first comment #4177 (comment) I can see you were able to deploy the app successfully and have route associated to it. Now all the routes which are created by openshift are by default exposed to port
80
(non-tls one) which is also expecting in your case and you were able tocurl http://ngins-nginx.apps-crc.testing
perform this.
I have a linux server(install crc) and a windows server, I want connect linux (crc nginx web) port "8080" from windows server. Now it works (I tested for weeks), thank anjannath, thanks all.
I still don't quite understand the real purpose of the Network mode (user or system).
On Linux you have the option, but on Windows and macOS only user mode is available.
System mode will use the system's provides virtual network to allow traffic from the VM to the Host and possible other nodes on your network.
For user-mode, this is replaced by an extra network stack, that communicates with the VM over virtual sockets and makes the traffic look like to originate from the localhost (or as local traffic).
The reason for this last option is to co-exist with the "Work from Home" scenario or in cases when a VPN is used in a route-all setup. This will otherwise disallow traffic other than local to be resolved. This would mean the 192.x/24
(or similar address) range would be forced to go over the company's VPN and proxy, however something like 127.1/32
would work (as this is local). If you have no need for using a VPN, or face other issues at the same time due to network configuration, do not use the user-mode on Linux, as systen might give you enough flexibility to set up a bridge or local only traffic. CRC does not enforce this set up on Linux after the VM has been created... we just need an active network connection that can pull container images.
will only work with user mode networking (ref)
This might be the case for 8080
as with user-mode
we will only claim the port on 127.0.0.1:8080
instead of the more broadly 0.0.0.0:8080
(all interfaces). However, this also means you run a local process on Linux which causes the conflict; something on your host machine already used this port, either by interface name or fixed IP.
I still don't quite understand the real purpose of the Network mode (user or system).
On Linux you have the option, but on Windows and macOS only user mode is available.
System mode will use the system's provides virtual network to allow traffic from the VM to the Host and possible other nodes on your network.
For user-mode, this is replaced by an extra network stack, that communicates with the VM over virtual sockets and makes the traffic look like to originate from the localhost (or as local traffic).
The reason for this last option is to co-exist with the "Work from Home" scenario or in cases when a VPN is used in a route-all setup. This will otherwise disallow traffic other than local to be resolved. This would mean the
192.x/24
(or similar address) range would be forced to go over the company's VPN and proxy, however something like127.1/32
would work (as this is local). If you have no need for using a VPN, or face other issues at the same time due to network configuration, do not use the user-mode on Linux, as systen might give you enough flexibility to set up a bridge or local only traffic. CRC does not enforce this set up on Linux after the VM has been created... we just need an active network connection that can pull container images.will only work with user mode networking (ref)
This might be the case for
8080
as withuser-mode
we will only claim the port on127.0.0.1:8080
instead of the more broadly0.0.0.0:8080
(all interfaces). However, this also means you run a local process on Linux which causes the conflict; something on your host machine already used this port, either by interface name or fixed IP.
Thank you very much for your explanation. I think the report issue has been resolved. Thank you.
General information
crc setup
before starting it (Yes/No)? YesCRC version
CRC version: 2.32.0+54a6f9 OpenShift version: 4.14.8 Podman version: 4.4.4
CRC status
CRC VM: Running OpenShift: Running (v4.14.8) RAM Usage: 19.43GB of 41.06GB Disk Usage: 57.18GB of 128.2GB (Inside the CRC VM) Cache Usage: 18.57GB Cache Directory: /root/.crc/cache
CRC config
Host Operating System
NAME="Red Hat Enterprise Linux" VERSION="9.2 (Plow)"
Steps to reproduce
apiVersion: route.openshift.io/v1 kind: Route metadata: name: nginx namespace: nginx annotations: openshift.io/host.generated: "true" labels: app: nginx app.kubernetes.io/component: nginx app.kubernetes.io/instance: nginx spec: host: ngins-nginx.apps-crc.testing to: kind: Service name: nginx weight: 100 port: targetPort: 8080-tcp wildcardPolicy: None
oc get all Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ NAME READY STATUS RESTARTS AGE pod/nginx-85958d65dd-kwhmg 1/1 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx ClusterIP 10.217.4.115 8080/TCP,8443/TCP 10m
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx 1/1 1 1 10m
NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-85958d65dd 1 1 1 10m
NAME IMAGE REPOSITORY TAGS UPDATED imagestream.image.openshift.io/bitnginx default-route-openshift-image-registry.apps-crc.testing/nginx/bitnginx latest 22 hours ago
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/nginx ngins-nginx.apps-crc.testing nginx 8080-tcp None