Containers cannot reach the external IP of their own server

[christianapel]

I have manually added the after.rules settings that are listed here and all containers that have not been exposed via ufw route allow are no longer reachable from external networks. The only container that is routed and reachable from outside is the web proxy, which is what I have expected so far.

One of the internal containers is providing an SSO API endpoint that is proxied and available on an external address https://openid.domain.com/auth/realms/domain.com/.well-known/openid-configuration. This endpoint must be called on the external address to return the proper results, which is also working with the above configuration.

But unfortunately, the other containers are not able to access the external API of the same server, while they can access e.g. google.com. Once they call the API on their own server and use its external address, the requests time out.

Do you have any idea about why this might not work?

[christianapel]

@chaifeng I would appreciate any hints... Thanks!

[chaifeng]

Hi @christianapel, sorry for I didn't reply to this issue.

Could containers access the SSO API normally before you applied the UFW rules?

Containers can access google.com which means DNS and NAT work. Could you try to run a ubuntu container and using dig openid.domain.com and curl -v htttps://openid.domain.com/auth to see that containers can resolve the domain name properly, and to check the IP address is private or public.

Is the SSO API endpoint proxied by the web proxy you mentioned?

If you have Vagrant and Virtualbox installed. You can clone this project and switch to branch add-bach-unit-testing-framework, and use vagrant up to create VMs. Using vagrant ssh master to log into the master node. If you can reproduce the problem in the VMs, send me the steps.

[chaifeng]

I don't know the details of your network and docker settings. But I think you can try to add one more rule.

iptables -A INPUT -i docker0 -j ACCEPT

[ozankose1992]

Hi @chaifeng,

First of all thank you for this great project. It was pretty informative/insightful for me. Updating the following rules as shown below fixes this problem for me. After checking out the logs of ufw I saw private IP addresses are blocked when you try to access your server with it's external IP (By the way, I am using docker-compose to run couple of containers).

-A DOCKER-USER -j ACCEPT -s 10.0.0.0/8 -A DOCKER-USER -j ACCEPT -s 172.16.0.0/12 -A DOCKER-USER -j ACCEPT -s 192.168.0.0/16

As far as I tested, the desired functionality is still achieved but I don't know if this update creates any security flaw. @chaifeng Do you have any comments?

[ozankose1992]

A quick update, after restarting and rebooting all the systems. I found out that the above solution does not work. Strangely while the sudo ufw route allow from 172.16.0.0/12 command changes nothing (still no access), sudo ufw allow from 172.16.0.0/12 gives access using external IP address.

[chaifeng]

@ozankose1992 hi, did you try iptables -A INPUT -i docker0 -j ACCEPT?

[ozankose1992]

@chaifeng yes I tried it but it didn't work for me. Logs of ufw says; it is blocking a request from 172.22.0.3 to IP address of the machine. So I had to explictly give allow permission to IP range 172.16.0.0/12 from ufw. (with ufw allow; ufw route allow does not work)

But I don't think it's something you need to fix/change. It may not be desirable to have default full access from private IP addresses to external IP address.

[slhck]

I have run into the same issue now. I think it's very much desirable to not prevent any outgoing access from a private internal network to the external IP of the same machine.

For example, different Docker micro-services that communicate amongst each other not via Docker networking, but through their external URL.

PS: I can confirm that this works:

ufw allow from 172.22.0.0/12

(For me the subnet was a different one… you have to find it out via docker network inspect)

[mitar]

Could this be added as default?

[ErikSteiner]

This seems to be my problem, too. I have two services in one container: pihole and dhcprelay. In order to use the DHCP service of pihole, I need to use a DHCP relay, that transfers the service from the docker network to my actual IP-range (192.XXX.XXX.1/24). The communication between those two is blocked by UFW. How can I (securely) achieve, that communication is permitted between those two?

Oct  7 21:30:36 kernel: [ 1453.125890] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=328 TOS=0x00 PREC=0x00 TTL=64 ID=11572 DF PROTO=UDP SPT=67 DPT=67 LEN=308 
Oct  7 21:30:38 kernel: [ 1454.882822] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:XX:XX:XX:XX:XX:XX:XX:88:ed:08:XX SRC=192.XXX.XXX.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=60588 DF PROTO=2 
Oct  7 21:30:38 kernel: [ 1455.062177] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=334 TOS=0x00 PREC=0x00 TTL=64 ID=11582 DF PROTO=UDP SPT=67 DPT=67 LEN=314 
Oct  7 21:30:38 kernel: [ 1455.081131] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=11585 DF PROTO=UDP SPT=67 DPT=67 LEN=324

I thought, that ufw-docker would help me with that issue. As a little insight, the composer file:

version: "3"

services:
  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    hostname: pihole
    ports:
      - "192.XXX.XXX.10:53:53/tcp"
      - "192.XXX.XXX.10:53:53/udp"
      - "80:80/tcp"
    environment:
      FTLCONF_LOCAL_IPV4: 192.XXX.XXX.10
      ...
    volumes:
       ...
    dns:
      - 127.0.0.1
    ...
    depends_on:
      - dhcprelay
    networks:
      npm-nw: {}
      pihole-nw:
        ipv4_address: 172.33.0.100

  dhcprelay:
    image: modem7/dhcprelay:latest
    container_name: isc-dhcp-relay
    hostname: isc-dhcp-relay
    command: ["-id", "eth0", "-iu", "br-pihole", "172.33.0.100"]
    ...
    network_mode: host
    ...

networks:
  pihole-nw:
    name: pihole-nw
    driver_opts:
      com.docker.network.bridge.name: br-pihole
    ipam:
      config:
        - subnet: 172.33.0.0/24
  npm-nw:
    name: npm-nw
    external: true

[chaifeng]

This seems to be my problem, too. I have two services in one container: pihole and dhcprelay. In order to use the DHCP service of pihole, I need to use a DHCP relay, that transfers the service from the docker network to my actual IP-range (192.XXX.XXX.1/24). The communication between those two is blocked by UFW. How can I (securely) achieve, that communication is permitted between those two?

Oct  7 21:30:36 kernel: [ 1453.125890] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=328 TOS=0x00 PREC=0x00 TTL=64 ID=11572 DF PROTO=UDP SPT=67 DPT=67 LEN=308 
Oct  7 21:30:38 kernel: [ 1454.882822] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:XX:XX:XX:XX:XX:XX:XX:88:ed:08:XX SRC=192.XXX.XXX.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=60588 DF PROTO=2 
Oct  7 21:30:38 kernel: [ 1455.062177] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=334 TOS=0x00 PREC=0x00 TTL=64 ID=11582 DF PROTO=UDP SPT=67 DPT=67 LEN=314 
Oct  7 21:30:38 kernel: [ 1455.081131] [UFW BLOCK] IN= OUT=br-pihole SRC=172.33.0.1 DST=172.33.0.100 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=11585 DF PROTO=UDP SPT=67 DPT=67 LEN=324

I thought, that ufw-docker would help me with that issue. As a little insight, the composer file:

version: "3"

services:
  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    hostname: pihole
    ports:
      - "192.XXX.XXX.10:53:53/tcp"
      - "192.XXX.XXX.10:53:53/udp"
      - "80:80/tcp"
    environment:
      FTLCONF_LOCAL_IPV4: 192.XXX.XXX.10
      ...
    volumes:
       ...
    dns:
      - 127.0.0.1
    ...
    depends_on:
      - dhcprelay
    networks:
      npm-nw: {}
      pihole-nw:
        ipv4_address: 172.33.0.100

  dhcprelay:
    image: modem7/dhcprelay:latest
    container_name: isc-dhcp-relay
    hostname: isc-dhcp-relay
    command: ["-id", "eth0", "-iu", "br-pihole", "172.33.0.100"]
    ...
    network_mode: host
    ...

networks:
  pihole-nw:
    name: pihole-nw
    driver_opts:
      com.docker.network.bridge.name: br-pihole
    ipam:
      config:
        - subnet: 172.33.0.0/24
  npm-nw:
    name: npm-nw
    external: true

Hi @ErikSteiner

Try

ufw route allow in eth0 proto igmp
ufw route allow out on br-pihole proto udp from any port 67 to any port 67

Please let me know if the above commands work :-P

[ErikSteiner]

ufw route allow out on br-pihole proto udp from any port 67 to any port 67 did not work. However, the following command worked: ufw allow out on br-pihole to any port bootps comment "dhcp to pihole bootps"