search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.4k stars 604 forks source link

Cannot connect to netcat server in rootful mode on exposed port from host (`nc -l` exits immediately) #1376

Open vsiravar opened 1 year ago

vsiravar commented 1 year ago

Description

When I try to connect to a netcat server running in a container through an exposed port from the host, I cannot connect to the server. This happens only in rootful mode.

Steps to reproduce

$ limactl shell default 
$ sudo systemctl restart containerd.service
$ sudo nerdctl run -d -p 6000:80 alpine sh -c "echo hello | nc -l -p 80"
bef8a45ec555fbda8768bf613a162138a62ec6062b245efedadc92dda0af32d9
$ sudo nerdctl ps -a
bef8a45ec555    docker.io/library/alpine:latest    "sh -c echo hello | …"    9 seconds ago     Exited (0) 5 seconds ago       0.0.0.0:6000->80/tcp    alpine-bef8a

nc server exits with exit code 0 without a client establishing a connection with it. Therefore running 

nc localhost 6000

does not connect to the server running in the container and therefore does not receive the "hello".

This behavior is only observed in rootful mode, increasing the timeout by running sudo nerdctl run -d -p 6000:80 alpine sh -c "echo hello | nc -l -p 80 -w 60" does not help either and the connection is closed before 60 seconds.

Expected behavior: Running sudo nerdctl run -d -p 6000:80 alpine sh -c "echo hello | nc -l -p 80" should not exit until a client establishes a connection.

Host info: 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:25 PDT 2022; root:xnu-8020.140.41~1/RELEASE_X86_64 x86_64

Lima version: 

lima -v
limactl version 0.12.1-rd1
ningziwen commented 1 year ago

Trying to use -it instead of -d

In Lima default VM,

$ sudo nerdctl run -it -p 6000:80 alpine sh
/ # echo hello
hello
/ # nc -l -p 80

Open a new terminal for lima default VM,

$ sudo nerdctl ps
CONTAINER ID    IMAGE                              COMMAND                   CREATED           STATUS    PORTS                   NAMES
d04ac59bfd05    docker.io/library/alpine:latest    "sh"                      20 minutes ago    Up        0.0.0.0:6000->80/tcp    alpine-d04ac

$ nc -v localhost 6000
nc: connect to localhost (127.0.0.1) port 6000 (tcp) failed: No route to host

$ sudo nerdctl exec d04ac59bfd05 nc -v localhost 80
FATA[0000] exec failed with exit code 1
ningziwen commented 1 year ago

Tried to start a python server with same alpine image and same port pair, proving that port forwarding works.

$ sudo nerdctl run -it -p 6000:80 alpine sh
/ # apk add --no-cache python3 py3-pip
/ # python3 -m http.server 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...

Another VM terminal:

$ curl localhost:6000
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>

The issue seems specifically for netcat.

Then tried nc again and it works.

/ # nc -l -p 80

$ nc -v localhost 6000
Connection to localhost (127.0.0.1) 6000 port [tcp/x11] succeeded!

This is intermittent error.

vsiravar commented 1 year ago

Adding another observable difference between rootless and rootful has been that netstat -an does not show listening sockets on the host when there is no server running on the container, however in rootless mode the port shows up. 

# rootless
limactl shell default  nerdctl run -d -p 10009:80 alpine sleep infinity 

# rootful
limactl shell default sudo nerdctl run -d -p 10008:80 alpine sleep infinity

# On the host 
# rootless
netstat -an | grep 10009
tcp4       0      0  127.0.0.1.10009        *.*                    LISTEN  

# Can connect although there is nothing running on port 80 on the container.
$ nc -v localhost 10009
Connection to localhost port 10009 [tcp/swdtp-sv] succeeded!

# rootful
# No socket is listening 
netstat -an | grep 10008

# Connection can't be established when there is nothing running on port 80 in on the container. 
$ nc -v localhost 10008
nc: connectx to localhost port 10008 (tcp) failed: Connection refused
nc: connectx to localhost port 10008 (tcp) failed: Connection refused

In both cases the containers are in running state. Could this possibly be due to implementation differences in port forwarding between rootless and rootful in nerdctl?

ningziwen commented 1 year ago

The issue is not specifically to netcat.

Built a image called pyalpine from Dockerfile

FROM alpine
RUN apk add --no-cache python3 py3-pip
$ sudo nerdctl run -d -p 6000:80 pyalpine sh -c "python3 -m http.server 80"

another terminal

$ curl localhost:6000
curl: (7) Failed to connect to localhost port 6000 after 1050 ms: No route to host

However, starting server later works.

$ sudo nerdctl run -d -p 6000:80 pyalpine sleep infinite
0a3bf4bb99d2b875ad196f786fab721e8607748bf8d74cd95dd12132cb75e841

$ sudo nerdctl exec 0a3bf4bb99d2b875ad196f786fab721e8607748bf8d74cd95dd12132cb75e841 sh -c "python3 -m http.server 80"
10.4.0.1 - - [24/Feb/2023 20:13:55] "GET / HTTP/1.1" 200 -

Another finding is the log behaviour is different for whether to start a server

$ sudo nerdctl run -d -p 6000:80 pyalpine sh -c "echo hello"
8ee7534981a4909146043b9bd3516ebb71aa4b1826c84d7dc428a035bdd8d92d
$ sudo nerdctl logs 8ee7534981a4909146043b9bd3516ebb71aa4b1826c84d7dc428a035bdd8d92d
hello

There is no log when starting a server.

$ sudo nerdctl run -d -p 6000:80 pyalpine sh -c "echo hello | python3 -m http.server 80"
35bdb417356ab592fdb89f52f8aa993bdccfe527ae368de7ad7f491350f22bbc
$ sudo nerdctl logs 35bdb417356ab592fdb89f52f8aa993bdccfe527ae368de7ad7f491350f22bbc

Same for sleep infinite.

$ sudo nerdctl run -d -p 6000:80 pyalpine sh -c "echo hello | sleep infinite"
46c01685533c4424d33ff0a4ce5e7d8904c79cddc81d081cd64dcd36294f7da6
$ sudo nerdctl logs 46c01685533c4424d33ff0a4ce5e7d8904c79cddc81d081cd64dcd36294f7da6

Update:

Can see logs by nerdctl log

$ sudo nerdctl run -d -p 6000:80 alpine sh -c "nc -l -v -p 80"
1f9a88e262ebf9904568f5fdb6008186db1a6c01a050f9f49fccb4fdbe37b9e1

$ sudo nerdctl logs 1f9a88e262ebf9904568f5fdb6008186db1a6c01a050f9f49fccb4fdbe37b9e1
listening on [::]:80 ...
connect to [::ffff:10.4.0.39]:80 from iad55-55-co-agg-r2-vl--803.amazon.com:52076 ([::ffff:10.4.0.1]:52076)