Connection refused between containers

myugan commented 3 years ago

HI @rodnymolina @ctalledo

A new issue for you guys hehe, probably I'm not sure what is the root cause of this problem, but I will explain what's going on in my machine. I was using sysbox with the latest version (deb package) and was found the issue between containers. I have 2 containers called A and B, both of them have SSH installed, and I want to SSH from container A to B. Still, I got Connection refused even though I check the SSH service is running or not from the host using docker exec -it container-a bash, and it was surprised me because the SSH service is running. This is my output result for the ip a show command asked by @rodnymolina when I talk to him.

Container A

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
8845: eth0@if8846: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.4/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Container B

Has installed Docker

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:48:16:3c:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: veth34965f8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether b2:cb:cf:f8:98:95 brd ff:ff:ff:ff:ff:ff link-netnsid 1
8833: eth0@if8834: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.16/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Host

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:cd:72:70:d4:93 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/20 brd 206.81.15.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.10.0.5/16 brd 10.10.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::cd:72ff:fe70:d493/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 4e:80:d1:93:5b:06 brd ff:ff:ff:ff:ff:ff
    inet 10.116.0.2/20 brd 10.116.15.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::4c80:d1ff:fe93:5b06/64 scope link
       valid_lft forever preferred_lft forever
4: br-325f920d83ff: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:b6:f8:29:68 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global br-325f920d83ff
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b6ff:fef8:2968/64 scope link
       valid_lft forever preferred_lft forever
2906: veth77b9881@if2905: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 66:50:1a:6a:3d:9a brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::6450:1aff:fe6a:3d9a/64 scope link
       valid_lft forever preferred_lft forever
2396: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:5a:d4:db:20 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 brd 172.20.255.255 scope global docker0
       valid_lft forever preferred_lft forever
2920: veth4219c47@if2919: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 52:47:f4:bc:fd:db brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::5047:f4ff:febc:fddb/64 scope link
       valid_lft forever preferred_lft forever
2922: veth83325dd@if2921: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 8e:f8:f3:32:bf:bb brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::8cf8:f3ff:fe32:bfbb/64 scope link
       valid_lft forever preferred_lft forever
8820: veth725c4f3@if8819: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 3e:a9:81:d7:3c:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3ca9:81ff:fed7:3c42/64 scope link
       valid_lft forever preferred_lft forever
8822: veth0fe942d@if8821: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 92:e2:db:41:b7:a9 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::90e2:dbff:fe41:b7a9/64 scope link
       valid_lft forever preferred_lft forever
8824: vethe5a5c42@if8823: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether da:b5:2f:bc:49:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::d8b5:2fff:febc:49d4/64 scope link
       valid_lft forever preferred_lft forever
8826: vethf6f9382@if8825: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 5e:7b:cc:7a:60:7a brd ff:ff:ff:ff:ff:ff link-netnsid 13
    inet6 fe80::5c7b:ccff:fe7a:607a/64 scope link
       valid_lft forever preferred_lft forever
8828: vetha5baf0c@if8827: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 2a:68:d0:7f:c3:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 15
    inet6 fe80::2868:d0ff:fe7f:c3e7/64 scope link
       valid_lft forever preferred_lft forever
8834: vethc362731@if8833: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 7a:c2:d8:d4:90:46 brd ff:ff:ff:ff:ff:ff link-netnsid 18
    inet6 fe80::78c2:d8ff:fed4:9046/64 scope link
       valid_lft forever preferred_lft forever
8836: veth8538e46@if8835: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether ae:dd:72:6a:23:77 brd ff:ff:ff:ff:ff:ff link-netnsid 17
    inet6 fe80::acdd:72ff:fe6a:2377/64 scope link
       valid_lft forever preferred_lft forever
8838: veth7e54f89@if8837: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 42:02:62:7b:ad:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 19
    inet6 fe80::4002:62ff:fe7b:ada8/64 scope link
       valid_lft forever preferred_lft forever
8840: vethf285e4a@if8839: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether 86:a5:50:24:ea:52 brd ff:ff:ff:ff:ff:ff link-netnsid 20
    inet6 fe80::84a5:50ff:fe24:ea52/64 scope link
       valid_lft forever preferred_lft forever
8846: veth4571798@if8845: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-325f920d83ff state UP group default
    link/ether ba:32:9f:59:d3:ac brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::b832:9fff:fe59:d3ac/64 scope link
       valid_lft forever preferred_lft forever

ctalledo commented 3 years ago

Hi @myugan , can you show us the docker run commands you are using the create the A & B containers?

rodnymolina commented 3 years ago

@myugan, the fact that you're using a custom docker-network (through iface: br-325f920d83ff) shouldn't have any impact on the internal ip traffic that flows through its associated bridge; as long as you use the internal ip addresses to reach your containers (in your case: 172.17/16 subnet) traffic shouldn't be dropped.

Can you please do the following?

Provide the ssh instruction that you're using within container-A to reach container-B.
Collect this output: docker network inspect <your-custom-docker-network>
Within both containers, get iptables output for both nat and filter tables: iptables -L -t filter && iptables -L -t nat

Also, in our previous slack conversation I believe you mentioned that problem was inconsistent. Is that right? That would be surprising for me.

rodnymolina commented 3 years ago

Concerning the potential network-overlapping issue that I previously raised (slack) after looking at your host's docker config, now I can see that this is not applicable to your setup as your connectivity problem is between sysbox (system) containers, and not between their inner containers. So I don't think you are dealing with a network overlapping issue here, specially since you have both containers connected through a dedicated custom-network.

rodnymolina commented 3 years ago

Closing this issue as it has not been reproduced ever since. Please reopen if that's not the case.

nestybox / sysbox

Connection refused between containers #274