Transparent mode using docker (option 1 and 2): `common.c:268:setsockopt IP_TRANSPARENT: Permission denied` or `0.0.0.0:443:bind: Permission denied`

LM1LC3N7 commented 10 months ago

Hello,

I am crawling this repo to find a solution for hours, but I found no answers to my case.

After trying both solutions 1 and 2 for deploying SSLH in transparent mode, and using docker, I always have an error:

Either it's common.c:268:setsockopt IP_TRANSPARENT: Permission denied (option 1) or 0.0.0.0:443:bind: Permission denied (option 2, and nothing is using this port). I even tried to add privileged: true, but still the same problem.

Few details about my server:

OS: Oracle Linux 8 (CentOS base, so)
docker version: 24.0.7
docker info: I am using user remap, but I specifically disable this configuration for SSLH container. So a root user in that container has real root rights on the host OS.
SELinux: enforced (and tested using "permissive" also)
SSLH version: both latest official image and local build

I don't know what to do, I am out of debug ideas 😢 What did I miss? It is a bug or just a misconfiguration from my side?

Thanks a lot for your time and help! 🙏

docker-compose.yml for transparent mode, option 1

version: '3'

services:

  # Service name
  sslh:
    #image: ghcr.io/yrutschle/sslh:latest
    build: https://github.com/yrutschle/sslh.git
    container_name: sslh
    restart: always
    hostname: sslh
    tty: true # does not work if removed
    # Config: https://github.com/yrutschle/sslh/blob/master/sslh.pod
    command: --transparent --foreground --listen=0.0.0.0:443 --tls=localhost:4443 --ssh=public-ip-redacted:22 --verbose-config=1 --verbose-probe-info=1 -n --on-timeout=tls
    environment:
      TZ: Europe/Paris
    ports:
      - 443:443/tcp
      - 4443:4443/tcp
    userns_mode: host # to disable the userns mode enabled by default
    security_opt: # try to disable SELinux just for debug purpose
      - label:disable
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE
    sysctls:
      - net.ipv4.conf.default.route_localnet=1
      - net.ipv4.conf.all.route_localnet=1
    extra_hosts:
      - localbox:host-gateway

When starting docker I have an error after few seconds:

 ✔ Container sslh  Started                                                                                                                                                                                                         0.1s
sslh  | --transparent flag is set
sslh  | Configuring iptables and routing...
sslh  | + iptables -t raw -A PREROUTING '!' -i lo -d 127.0.0.0/8 -j DROP
sslh  | + iptables -t mangle -A POSTROUTING '!' -o lo -s 127.0.0.0/8 -j DROP
sslh  | + iptables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh  | + iptables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh  | + ip rule add fwmark 0x1 lookup 100
sslh  | + ip route add local 0.0.0.0/0 dev lo table 100
sslh  | + cat /proc/sys/net/ipv6/conf/all/disable_ipv6
sslh  | + '[' 1 -eq 0 ]
sslh  | + set -e
sslh  | + set +x
sslh  | Executing with user 'sslh': sslh --transparent --foreground --listen=0.0.0.0:443 --tls=localhost:4443 --ssh=public-ip-redacted:22 --verbose-config=1 --verbose-probe-info=1 -n --on-timeout=tls
sslh  | ssh addr: public-ip-redacted:22 family 2 2. libwrap service: (null) log_level: 1 [] [fork] []
sslh  | tls addr: 127.0.0.1:4443 family 2 2. libwrap service: (null) log_level: 1 [] [] []
sslh  | timeout: 5
sslh  | on-timeout: tls
sslh  | UDP hash size: 1024
sslh  | Listening to:
sslh  | 3:      0.0.0.0:443     [] []
sslh  | Landlock: not built in
sslh  | sslh-select head-2024-01-16 started
sslh  | common.c:268:setsockopt IP_TRANSPARENT: Permission denied
sslh exited with code 0

docker-compose.yml for transparent mode, option 2

# must be set manually on the host:
# sysctl -w net.ipv4.conf.default.route_localnet=1
# sysctl -w net.ipv4.conf.all.route_localnet=1
# modprobe ip6table_mangle # (to solve another issue I got)

version: '3'

services:

  # Service name
  sslh:
    #image: ghcr.io/yrutschle/sslh:latest
    build: https://github.com/yrutschle/sslh.git
    container_name: sslh
    restart: always
    tty: true
    # Config: https://github.com/yrutschle/sslh/blob/master/sslh.pod
    command: --transparent --foreground --listen=0.0.0.0:443 --tls=localhost:4443 --ssh=public-ip-redacted:22 --verbose-config=1 --verbose-probe-info=1 -n --on-timeout=tls
    userns_mode: host
    network_mode: host
    security_opt:
      - label:disable
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE

Error logs for binding the port 443:

sslh  | + ip rule add fwmark 0x1 lookup 100
sslh  | + ip route add local 0.0.0.0/0 dev lo table 100
sslh  | ip: RTNETLINK answers: File exists
sslh  | + cat /proc/sys/net/ipv6/conf/all/disable_ipv6
sslh  | + '[' 0 -eq 0 ]
sslh  | + ip6tables -t raw -A PREROUTING '!' -i lo -d ::1/128 -j DROP
sslh  | + ip6tables -t mangle -A POSTROUTING '!' -o lo -s ::1/128 -j DROP
sslh  | + ip6tables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh  | + ip6tables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh  | + ip -6 rule add fwmark 0x1 lookup 100
sslh  | + ip -6 route add local ::/0 dev lo table 100
sslh  | ip: RTNETLINK answers: File exists
sslh  | + set -e
sslh  | + set +x
sslh  | Executing with user 'sslh': sslh --transparent --foreground --listen=0.0.0.0:443 --tls=localhost:4443 --ssh=public-ip-redacted:22 --verbose-config=1 --verbose-probe-info=1 -n --on-timeout=tls
sslh  | ssh addr: public-ip-redacted:22 family 2 2. libwrap service: (null) log_level: 1 [] [fork] []
sslh  | tls addr: ::1:4443 family 10 10. libwrap service: (null) log_level: 1 [] [] []
sslh  | timeout: 5
sslh  | on-timeout: tls
sslh  | UDP hash size: 1024
sslh  | Listening to:
sslh  | 0.0.0.0:443:bind: Permission denied
sslh exited with code 1

yrutschle commented 4 months ago

Did you give sslh the appropriate CAP_NET_RAW and CAP_NET_BIND_SERVICE capabilities?

LM1LC3N7 commented 4 months ago

Yes I did. Both capabilities. This is why I opened an issue. I am using Oracle Linux 8, with nftable (default on multiple Centos based distro) and docker. Do you have any other idea to debug? Thanks 🙏

yrutschle commented 4 months ago

Yes: run sslh with verbose-config: 1, and check it really does have the capabilities; if I understand correctly, you set them in the docker, which means the docker has the ability to have the capabilities, but does not mean sslh has them. If it doesn't, you need to setcap the appropriate binary. It's also worth trying outside of a docker, I guess.

LM1LC3N7 commented 4 months ago

Still same error:

sslh-t02  | --transparent flag is set
sslh-t02  | Configuring iptables and routing...
sslh-t02  | + iptables -t raw -A PREROUTING '!' -i lo -d 127.0.0.0/8 -j DROP
sslh-t02  | + iptables -t mangle -A POSTROUTING '!' -o lo -s 127.0.0.0/8 -j DROP
sslh-t02  | + iptables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh-t02  | + iptables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh-t02  | + ip rule add fwmark 0x1 lookup 100
sslh-t02  | + ip route add local 0.0.0.0/0 dev lo table 100
sslh-t02  | ip: RTNETLINK answers: File exists
sslh-t02  | + cat /proc/sys/net/ipv6/conf/all/disable_ipv6
sslh-t02  | + '[' 0 -eq 0 ]
sslh-t02  | + ip6tables -t raw -A PREROUTING '!' -i lo -d ::1/128 -j DROP
sslh-t02  | + ip6tables -t mangle -A POSTROUTING '!' -o lo -s ::1/128 -j DROP
sslh-t02  | + ip6tables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh-t02  | + ip6tables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh-t02  | + ip -6 rule add fwmark 0x1 lookup 100
sslh-t02  | + ip -6 route add local ::/0 dev lo table 100
sslh-t02  | ip: RTNETLINK answers: File exists
sslh-t02  | + set -e
sslh-t02  | + set +x
sslh-t02  | Executing with user 'sslh': sslh --verbose-config=1 --transparent --foreground --listen=0.0.0.0:443 --tls=localhost:4443 --ssh=REDACTED:22 --verbose-probe-info=1 -n --on-timeout=tls
sslh-t02  | ssh addr: REDACTED:22 family 2 2. libwrap service: (null) log_level: 1 [] [fork] []
sslh-t02  | tls addr: ::1:4443 family 10 10. libwrap service: (null) log_level: 1 [] [] []
sslh-t02  | timeout: 5
sslh-t02  | on-timeout: tls
sslh-t02  | UDP hash size: 1024
sslh-t02  | Listening to:
sslh-t02  | 0.0.0.0:443:bind: Permission denied
sslh-t02 exited with code 0

docker-compose.yml

services:
  sslh:
    build: https://github.com/yrutschle/sslh.git
    container_name: sslh-t02
    environment:
      TZ: Europe/Paris
    restart: always
    tty: true
    command: >
        --verbose-config=1
        --transparent
        --foreground
        --listen=0.0.0.0:443
        --tls=localhost:4443
        --ssh=REDACTED:22
        --verbose-probe-info=1
        -n
        --on-timeout=tls
    # privileged: true
    user: root # To test, should not be needed
    userns_mode: host
    network_mode: host
    security_opt:
      - label:disable # To test, should not be needed
      - no-new-privileges=false # To test, should not be needed
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE

Nothing is using the 443 port:

$ sudo netstat -an | grep ":443"
tcp        0      0 127.0.0.1:44321         0.0.0.0:*               LISTEN
tcp        0      0 REDACTED:59796    REDACTED:443       ESTABLISHED
tcp6       0      0 ::1:44321               :::*                    LISTEN

LM1LC3N7 commented 4 months ago

[EDIT: REMOVED]

LM1LC3N7 commented 4 months ago

Found the issue:

using the last "master" docker image
by default my docker configuration prevent containers to gain more privileges. Adding solve the permission denied.

Full docker-compose.yml

services:
  sslh:
    image: ghcr.io/yrutschle/sslh:master
    container_name: sslh
    environment:
      TZ: Europe/Paris
    restart: always
    tty: true
    # Config: https://github.com/yrutschle/sslh/blob/master/sslh.pod
    command: >
        --verbose-config=1
        --transparent
        --foreground
        --listen=0.0.0.0:443
        --tls=localhost:4443
        --ssh=REDACTED:22
        --verbose-probe-info=1
        -n
        --on-timeout=tls
    userns_mode: host
    network_mode: host
    security_opt:
      - no-new-privileges=false
    # Drop capabilities (man 7 capabilities for list)
    # Should drop ALL by default
    # https://dockerlabs.collabnix.com/advanced/security/capabilities/
    cap_drop:
      - ALL
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE
      - DAC_OVERRIDE
      - SETGID
      - SETUID

LM1LC3N7 commented 4 months ago

Issue solved, as I don't have anymore a bind error.

I must have an iptables issue somewhere: from internet I can't connect to https services (connection reset), but from the host usint cURL, traefik handle correctly the request (through sslh).🤔

LM1LC3N7 commented 4 months ago

According to your last discussion about going transparent mode (https://github.com/yrutschle/sslh/discussions/454), I tried the solution (https://github.com/yrutschle/sslh/blob/master/doc/simple_transparent_proxy.md).

On CentOS 8 (Oracle Linux), it is slightly different:

Create the new dummy interface nmcli connection add type dummy ifname dummy0 con-name dummy0 ip4 192.168.255.254/32
Add commands for up/down: sudo vi /etc/NetworkManager/dispatcher.d/10-dummy0-routes

#!/bin/bash

if [ "$1" == "dummy0" ]; then
case "$2" in
    up)
        ip rule add from 192.168.255.254 table sslh
        ip route add local 0.0.0.0/0 dev dummy0 table sslh
        ;;
    down)
        ip route del local 0.0.0.0/0 dev dummy0 table sslh
        ip rule del from 192.168.255.254 table sslh
        ;;
esac
fi

Apply rights and remove old rules:

sudo chmod +x /etc/NetworkManager/dispatcher.d/10-dummy0-routes
sudo systemctl restart NetworkManager
sudo sysctl -w net.ipv4.conf.default.route_localnet=0
sudo sysctl -w net.ipv4.conf.all.route_localnet=0

Start the network card with mcli connection up dummy0

The container now starts but packets are not routed when coming from internet or localhost:

sslh-t02  | --transparent flag is set
sslh-t02  | Configuring iptables and routing...
sslh-t02  | + iptables -t raw -A PREROUTING '!' -i lo -d 127.0.0.0/8 -j DROP
sslh-t02  | + iptables -t mangle -A POSTROUTING '!' -o lo -s 127.0.0.0/8 -j DROP
sslh-t02  | + iptables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh-t02  | + iptables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh-t02  | + ip rule add fwmark 0x1 lookup 100
sslh-t02  | + ip route add local 0.0.0.0/0 dev lo table 100
sslh-t02  | + cat /proc/sys/net/ipv6/conf/all/disable_ipv6
sslh-t02  | + '[' 0 -eq 0 ]
sslh-t02  | + ip6tables -t raw -A PREROUTING '!' -i lo -d ::1/128 -j DROP
sslh-t02  | + ip6tables -t mangle -A POSTROUTING '!' -o lo -s ::1/128 -j DROP
sslh-t02  | + ip6tables -t nat -A OUTPUT -m owner --uid-owner sslh -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j CONNMARK --set-xmark 0x01/0x0f
sslh-t02  | + ip6tables -t mangle -A OUTPUT '!' -o lo -p tcp -m connmark --mark 0x01/0x0f -j CONNMARK --restore-mark --mask 0x0f
sslh-t02  | + ip -6 rule add fwmark 0x1 lookup 100
sslh-t02  | + ip -6 route add local ::/0 dev lo table 100
sslh-t02  | + set -e
sslh-t02  | + set +x
sslh-t02  | Executing with user 'sslh': sslh --verbose-config=1 --transparent --foreground --user sslh --listen=PUB_IP:443 --tls=192.168.255.254:64561 --ssh=192.168.255.254:22 --verbose-probe-info=1 --on-timeout=tls
sslh-t02  | ssh addr: 192.168.255.254:ssh family 2 2. libwrap service: (null) log_level: 1 [] [fork] []
sslh-t02  | tls addr: 192.168.255.254:64561 family 2 2. libwrap service: (null) log_level: 1 [] [] []
sslh-t02  | timeout: 5
sslh-t02  | on-timeout: tls
sslh-t02  | UDP hash size: 1024
sslh-t02  | Listening to:
sslh-t02  | 3:  PUB_IP:https        [] []
sslh-t02  | turning into sslh
sslh-t02  | Landlock: not built in
sslh-t02  | sslh-select head-2024-06-26 started
sslh-t02  | probing for ssh
sslh-t02  | probed for ssh: PROBE_NEXT
sslh-t02  | probing for tls
sslh-t02  | probed for tls: PROBE_MATCH
sslh-t02  | tls: lost incoming connection

LM1LC3N7 commented 4 months ago

Currently I am stuck at this point @yrutschle, I don't know what to do.

To summarize:

First main issue was error with 443 port binding, solved by using the :master docker image
Second issue is no transparent mode working, sslh is starting but no traffic is permit except from localhost using the "historical" transparent configuration. With the new solution, same issue, but also local traffic does not work.

docker-compose.yml

services:
  sslh:
    image: ghcr.io/yrutschle/sslh:master
    container_name: sslh
    environment:
      TZ: Europe/Paris
    restart: always
    tty: true
    # Config: https://github.com/yrutschle/sslh/blob/master/sslh.pod
    command: >
        --verbose-config=1
        --transparent
        --foreground
        --user=sslh
        --listen=PUB_IP:443
        --tls=192.168.255.254:64561
        --ssh=192.168.255.254:22
        --verbose-probe-info=1
        --on-timeout=tls
    -n
    userns_mode: host
    network_mode: host
    security_opt:
      - no-new-privileges=false
    # Drop capabilities (man 7 capabilities for list)
    # Should drop ALL by default
    # https://dockerlabs.collabnix.com/advanced/security/capabilities/
    cap_drop:
      - ALL
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE
      - DAC_OVERRIDE
      - SETGID
      - SETUID

ftasnetamot commented 2 months ago

Just seeing this issue. You are merging two methods of transparency. In your interface configuration you prepare for the iproute2-only solution with dummy-interface, but in your container log the old iptables-marking method and 127.0.0.1 is used. So make sure, that you really use either the old or the new method. Check all your configurations, that things really fit together.

This said, as the new iproute2-only configuration option is currently not included in almost all recipes or packages.

And: If you wish to use IPV6, you need a unique IPV6 address on dummy0 as well, and the according routing rules for the new recipe.

LM1LC3N7 commented 2 months ago

Thanks a lot for your time and answer. I will check all again on my new server as soon as I can :)

yrutschle / sslh

Transparent mode using docker (option 1 and 2): `common.c:268:setsockopt IP_TRANSPARENT: Permission denied` or `0.0.0.0:443:bind: Permission denied` #424

docker-compose.yml for transparent mode, option 1

docker-compose.yml for transparent mode, option 2