search

greenpau / cni-plugins

CNI Plugins compatible with nftables
Apache License 2.0
48 stars 13 forks source link

No access to localhost ports #4

Closed ivanov17 closed 2 years ago

ivanov17 commented 3 years ago

As I previously wrote, I have been trying to use podman on CentOS 8 running on qemu/kvm virtual machine. I removed firewalld and I use nftables directly now.

Unfortunately, I have some troubles with my nftables rules and an issue related to access to localhost ports inside the virtual machine.

When I start nginx container and try to access to localhost on 80 port, the connection hangs until I interrupt it:

# curl -vvv localhost
* Rebuilt URL to: localhost/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.61.1
> Accept: */*
>

It also happens if I try to access 80 port using ip address of my virtual machine inside the VM:

# curl -vvv -4 192.168.122.101
* Rebuilt URL to: 192.168.122.101/
*   Trying 192.168.122.101...
* TCP_NODELAY set
* Connected to 192.168.122.101 (192.168.122.101) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.122.101
> User-Agent: curl/7.61.1
> Accept: */*
>

Outside the VM, I can access the container's port without any problem (If I remove the reject rule from the forward chain of the inet filter table, of course):

$ curl 192.168.122.101
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

When I stop nginx container, install package with another webserver on the virtual machine and then try to access 80 port on localhost inside the VM, I reach it successfully:

# curl localhost
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
  <title>Powered by lighttpd</title>
  <link rel="shortcut icon" href="favicon.ico" />
  <link rel="icon" href="favicon.ico" />
  <style type="text/css">
    <!--
      body { background: white; color: #666f85; text-align: center }
      img  { border: none }
    -->
  </style>
</head>
<body>

<p>
<img src="light_logo.png" width="249" height="239" alt="LIGHTTPD - fly light." />
</p>

<p>
<img src="poweredby.png" width="88" height="31" alt="Powered by GNU/Linux" />
<br />
<a href="http://www.lighttpd.net/"><img src="light_button.png" width="80" height="27" alt="Powered by Lighttpd" /></a>
</p>

</body>
</html>

I didn't create another CentOS 8 VM with default configuration running firewalld to test the access to localhost ports. But I use firewalld and podman in Fedora 32 on my home machine and I can access 80 port on localhost running the same nginx container.

It looks like there is an issue with rules set by CNI plugins.

This is a listing of my nftables ruleset when container is running:

# nft list ruleset
table inet filter {
    chain input {
        type filter hook input priority filter; policy accept;
        ct state established,related counter packets 86867 bytes 92621887 accept
        meta l4proto { icmp, ipv6-icmp } counter packets 4173 bytes 305844 accept
        iif "lo" counter packets 2746 bytes 285833 accept
        ip6 saddr fe80:: ip6 daddr fe80:: udp sport 547 udp dport 546 ct state new counter packets 0 bytes 0 accept
        tcp dport 22 ct state new counter packets 0 bytes 0 accept
        tcp dport 80 ct state new counter packets 15 bytes 900 accept
        tcp dport 443 ct state new counter packets 0 bytes 0 accept
        counter packets 1951 bytes 346572 reject with icmpx type admin-prohibited
    }

    chain forward {
        type filter hook forward priority filter; policy accept;
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        jump cni-npo-50795b7a600c9b95c61de4b
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        jump cni-npr-50795b7a600c9b95c61de4b
    }

    chain output {
        type nat hook output priority -100; policy accept;
    }

    chain input {
        type nat hook input priority 100; policy accept;
    }

    chain cni-npr-50795b7a600c9b95c61de4b {
        iifname != "cni-podman0" tcp dport 80 dnat to 10.88.2.129:80
    }

    chain cni-npo-50795b7a600c9b95c61de4b {
        iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return
        iifname "cni-podman0" ip saddr 10.88.2.129 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
        iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 0 bytes 0 masquerade
    }
}
table ip raw {
    chain prerouting {
        type filter hook prerouting priority raw; policy accept;
    }
}
table ip filter {
    chain forward {
        type filter hook forward priority filter; policy drop;
        jump cni-ffw-50795b7a600c9b95c61de4b
        oifname "cni-podman0" ip daddr 10.88.2.129 tcp dport 80 counter packets 3 bytes 180 accept
        log prefix "ip4 forward drop: "
        counter packets 0 bytes 0 drop
    }

    chain cni-ffw-50795b7a600c9b95c61de4b {
        oifname "cni-podman0" ip daddr 10.88.2.129 ct state established,related counter packets 30 bytes 3106 accept
        iifname "cni-podman0" ip saddr 10.88.2.129 counter packets 27 bytes 4594 accept
        iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
    }
}
table ip6 nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        jump cni-npo-50795b7a600c9b95c61de4b
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        jump cni-npr-50795b7a600c9b95c61de4b
    }

    chain output {
        type nat hook output priority -100; policy accept;
    }

    chain input {
        type nat hook input priority 100; policy accept;
    }

    chain cni-npr-50795b7a600c9b95c61de4b {
        iifname != "cni-podman0" tcp dport 80 dnat to [fd10:88:7334:6d55::1e]:80
    }

    chain cni-npo-50795b7a600c9b95c61de4b {
        iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e ip6 daddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 return
    }
}
table ip6 raw {
    chain prerouting {
        type filter hook prerouting priority raw; policy accept;
    }
}
table ip6 filter {
    chain forward {
        type filter hook forward priority filter; policy drop;
        jump cni-ffw-50795b7a600c9b95c61de4b
        oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e tcp dport 80 counter packets 0 bytes 0 accept
        log prefix "ip6 forward drop: "
        counter packets 0 bytes 0 drop
    }

    chain cni-ffw-50795b7a600c9b95c61de4b {
        oifname "cni-podman0" ip6 daddr fd10:88:7334:6d55::1e ct state established,related counter packets 0 bytes 0 accept
        iifname "cni-podman0" ip6 saddr fd10:88:7334:6d55::1e counter packets 0 bytes 0 accept
        iifname "cni-podman0" oifname "cni-podman0" counter packets 0 bytes 0 accept
    }
}

This is my CNI configuration:

# podman network inspect podman
[
    {
        "cniVersion": "0.4.0",
        "name": "podman",
        "plugins": [
            {
                "bridge": "cni-podman0",
                "hairpinMode": true,
                "ipMasq": false,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "10.88.0.1",
                                "subnet": "10.88.0.0/16"
                            }
                        ],
                        [
                            {
                                "gateway": "fd10:88:7334:6d55::1",
                                "subnet": "fd10:88:7334:6d55::/64"
                            }
                        ]
                    ]
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        },
                        {
                            "dst": "::/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "cni-nftables-portmap"
            },
            {
                "forward_chain_name": "forward",
                "type": "cni-nftables-firewall"
            },
            {
                "type": "tuning"
            }
        ]
    }
]

Podman version and system info:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64
# podman system info
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: bc11c2e39bf61429aafb5d131e709a970036be24'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 1064951808
  MemTotal: 1915486208
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc10.module_el8.2.0+304+65a3c2ac.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 1073737728
  SwapTotal: 1073737728
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: centos8.lan
  kernel: 4.18.0-193.19.1.el8_2.x86_64
  os: linux
  rootless: false
  uptime: 57h 7m 31.46s (Approximately 2.38 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
  - registry.access.redhat.com
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes
greenpau commented 3 years ago

@ivanov17 , can you access container IP address from host namespace?

greenpau commented 3 years ago

@ivanov17 , generally speaking ... if you try accessing localhost from host namespace hoping getting to container, I don't think it would work, because there are no rules accomplishing that. There is some work left "to do" with this plugin.

mebel-christ commented 3 years ago

+1\ I am facing the same issue (for obvious reasons). This is kind of bugging me even so basically everything works fine it would be great to access the containers via localhost.\ @greenpau do you plan to implement that?

greenpau commented 3 years ago

@micaeb , did not have a chance to look at it. Hoping someone else would fix it :-)

ghost commented 3 years ago

The problem is that prerouting hook will trigger only for packets from real device or bridge and won`t work for outgoing packages from local machine. For details see https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks. It is possible to do local DNAT as DNAT statements will work in output hook.

greenpau commented 3 years ago

@ivanov17 , please check whether it work or not with the latest release. If it does, please close the issue.