Containers cannot reach eachother (PhotonOS 3, minimal, clean install).

[number4jazz]

Hi guys. spent several days on resolving this. But need your help to solve my issue.

Description: On a clean system install, I started Docker and run several containers (Portainer, Influxdb, Chronograf, Telegraf).

Connectivity to the containers from the outside (e.g. a different computer) works perfectly. Connectivity from the containers to the outside works also perfectly. But the containers on the same dockerhost cannot reach eachother.

Reflection So, my system setup runs fine as long as I do not interconnect the containers (e.g. run telegraf on a different server), but ofcourse, this is not the objective of having a dedicated VM as dockerhost.

My personal thoughts: PhotonOS somehow restrict network traffic from one container to another.

The exact (!) same docker setup works fine when run on Centos8 (both bare-metal, as in VM). I made a picture of the three situation that I tested with.

System description 10th generation i7 NUC (NUCi7FNH, 64 GB ram) Clean install of Centos8 (minimal, SElinux disabled, firewalld disabled, root access) Clean install of VMware Workstation pro 16 (Linux version, nogui) Clean install of VM with PhotonOS (minimal, 8 GB ram) .. it is the only VM running on the VMware host / I tried both the ISO and the OVA alternative. VMware network adapter in bridge mode. Containers network run in bridge mode.

Any help, experiences, best practices? Would be really appreciated. Thanks.

greetings, Jasper.

[number4jazz]

Ok, so I found a thing... my problem is solved when using Photon OS 4 beta.

In order to find differences in networking .. I installed 2 fresh Photon OS instances (OVA hw13): 1 version 3, and 1 version 4.

The only thing after logging in is to start docker (systemctl start docker) and to show the iptables (iptables-save).

What surprises me is that both iptables are quiet different.. especially the *mangle section in version 3.

So .. what I can do is switch to version 4 (and become a beta tester yeah :-) ... but is there anyone how can confirm that something is broken in version 3? I figure someone else experienced the same issue before?

the iptables-save from version 3

# Generated by iptables-save v1.8.3 on Mon Dec 21 22:16:48 2020
*mangle
:PREROUTING ACCEPT [855:107904]
:INPUT ACCEPT [853:107248]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [476:68775]
:POSTROUTING ACCEPT [476:68775]
COMMIT
# Completed on Mon Dec 21 22:16:48 2020
# Generated by iptables-save v1.8.3 on Mon Dec 21 22:16:48 2020
*nat
:PREROUTING ACCEPT [291:57598]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [3:228]
:POSTROUTING ACCEPT [3:228]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Dec 21 22:16:48 2020
# Generated by iptables-save v1.8.3 on Mon Dec 21 22:16:48 2020
*filter
:INPUT DROP [291:57598]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Dec 21 22:16:48 2020

the iptables-save from version 4

# Generated by iptables-save v1.8.4 on Mon Dec 21 22:17:08 2020
*filter
:INPUT ACCEPT [612:82639]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [319:45156]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Dec 21 22:17:08 2020
# Generated by iptables-save v1.8.4 on Mon Dec 21 22:17:08 2020
*nat
:PREROUTING ACCEPT [301:59717]
:INPUT ACCEPT [301:59717]
:OUTPUT ACCEPT [5:420]
:POSTROUTING ACCEPT [5:420]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Mon Dec 21 22:17:08 2020

[benwa]

I think I'm having the same issue in PhotonOS 4.0 1526e30ba.

I'm using a brand new OVA and netbox-docker. The netbox image can't seem to connect to the postgres one and netbox-worker can't connect to redis.

root@netbox [ ~/netbox-docker]# iptables-save
# Generated by iptables-save v1.8.7 on Thu Aug 12 01:26:44 2021
*filter
:INPUT ACCEPT [1133:78230]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [783:93635]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-59c125505876 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-59c125505876 -j DOCKER
-A FORWARD -i br-59c125505876 ! -o br-59c125505876 -j ACCEPT
-A FORWARD -i br-59c125505876 -o br-59c125505876 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.18.0.6/32 ! -i br-59c125505876 -o br-59c125505876 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-59c125505876 ! -o br-59c125505876 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-59c125505876 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Aug 12 01:26:44 2021
# Generated by iptables-save v1.8.7 on Thu Aug 12 01:26:44 2021
*nat
:PREROUTING ACCEPT [288:27333]
:INPUT ACCEPT [7:252]
:OUTPUT ACCEPT [11:826]
:POSTROUTING ACCEPT [11:826]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.18.0.0/16 ! -o br-59c125505876 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.6/32 -d 172.18.0.6/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A DOCKER -i br-59c125505876 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-59c125505876 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.6:8080
COMMIT
# Completed on Thu Aug 12 01:26:44 2021

[benwa]

I've also seen another person (I think it's @AllanVarkey) have the same problem here: https://unix.stackexchange.com/questions/663511/cant-connect-to-docker-containers-running-on-photon-os-4-0-linux-server

[atilllla2]

Is there any workaround for this? photon 4.0 rev2 still has this problem. Basically useless in this state for docker.

[dcasota]

Hi @atilllla2 would you mind to hang in here to a micro part of this issue and share some findings?

The attached vanila service testyml.txt (-> rename to test.yml) might be a helpful starting point.

docker swarm init
docker stack deploy -c test.yml test

I've used a different combo virtualization infrastructure + Photon OS flavor (Ph4 Azure .vhd setup). Before considering the situation on Photon OS 3 minimal it could be helpful to have a Inner-PhotonOS4rev2-centric-solution with container secure-intercommunication (ssl) findings as well.

Well, the docker service starts, tested on Ph4rev2.

root@ph4 [ ~ ]# docker service ls
ID             NAME              MODE         REPLICAS   IMAGE                       PORTS
8zuxgp6y9xfw   test_chronograf   replicated   1/1        chronograf:1.9              *:8888->8888/tcp
k0nmcplr2lqz   test_grafana      replicated   1/1        grafana/grafana-oss:8.5.2   *:3000->3000/tcp
31fzf8qgu00a   test_influxdb     replicated   1/1        influxdb:2.2                *:8086->8086/tcp
ioe4ght86ge0   test_mqtt         replicated   1/1        eclipse-mosquitto:2.0.14    *:1883->1883/tcp, *:9001->9001/tcp
vkikxqiaoiha   test_telegraf     replicated   1/1        telegraf:1.20               *:8094->8094/tcp, *:8092->8092/udp, *:8125->8125/udp
root@ph4 [ ~ ]# 

Can you describe what to modify/test? I'm not familiar with the components at all. For sure the specified docker image flavors, servers config+ports,etc. must be corrected to get a sort of functioning solution (desired state with green arrows).

[atilllla2]

In brige mode (default) Each docker container that you see has an internal ip like 172.17.x.x This ips are dinamic ips. For example if you want to reach your db from grafana you have to set the db ip in grafana. Solution 1: you can set the internal ip. It will work. But it will be changed with each restart. Solution 2 you can set the host ip:port number. You can reach this way your container from everywhere but from the neighbour container. This is the problem. Problem 2: when you use docker containers in host mode, you cannot reach the container.

[atilllla2]

[atilllla2]

I think the problem is the same as here. Iptables turning off then everything works. So we need to set iptables according to this. https://github.com/vmware/photon/issues/1321

[atilllla2]

I just forgot to put here that my part is solved formelly modifying iptables like this.

iptables -A INPUT -m iprange --src-range 172.17.1.1-172.17.1.255 -j ACCEPT iptables-save > /etc/systemd/scripts/ip4save