BastilleBSD / bastille

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.
https://bastillebsd.org
BSD 3-Clause "New" or "Revised" License
823 stars 130 forks source link

[BUG] Fresh install on FreeBSD 14.1: jails cannot access internet via the host #703

Open waseigo opened 3 months ago

waseigo commented 3 months ago

[MANDATORY] Describe the bug [MANDATORY] Bastille was running just fine on a FreeBSD 14.0 host. I upgraded the host to 14.1 and all jails lost access to the internet. To verify that this was not something specific to that one host, I tried again with a fresh install of bastille on a fresh 14.1 install on a VM. Same behavior.

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output)

0.10.20231125
14.1-RELEASE
14.1-RELEASE
14.1-RELEASE

[MANDATORY] How did you install bastille? (port/pkg/git)

pkg

[optional] Steps to reproduce?

  1. Install FreeBSD 14.1 or upgrade existing host to 14.1.
  2. Install bastille from pkg/latest.
  3. Follow bastille's Getting Started guide, including pf.conf.
  4. Create new jail.
  5. Attempt to pkg install ....
  6. See below. Host resolution fails.
root@gandalf:~ # bastille list
   JID  IP Address      Hostname                      Path
     2  172.16.33.90    minio                         /usr/local/bastille/jails/minio/root
root@gandalf:~ # bastille pkg minio install -y htop minio
[minio]:
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly.
Consider changing PACKAGESITE.

The problem persists after a reboot (just to make sure that eveyrthing that's supposed to run also runs at boot.

user@gandalf:~ $ doas bastille pkg 
Usage: bastille pkg [-H|--host] TARGET command [args]
user@gandalf:~ $ doas bastille pkg minio install minio
[minio]:
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly, please wait...
pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly/Latest/pkg.txz: Host does not resolve
Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/quarterly.
Consider changing PACKAGESITE.

Failed to apply on some jails, please check logs

[optional] Expected behavior

Host resolution succeeds.

[optional] Other information

uname -a

root@gandalf:~ # uname -a 
FreeBSD gandalf 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64

/etc/rc.conf

root@gandalf:~ # cat /etc/rc.conf
hostname="gandalf"
ifconfig_vtnet0="DHCP"
local_unbound_enable="YES"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
bastille_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
pf_enable="YES"

ifconfig

root@gandalf:~ # ifconfig
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether 52:54:00:07:a5:32
    inet 192.168.88.6 netmask 0xffffff00 broadcast 192.168.88.255
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 172.16.33.90 netmask 0xffffff00
    inet6 fe80::1%bastille0 prefixlen 64 scopeid 0x3
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

/etc/pf.conf

root@gandalf:~ # cat /etc/pf.conf 
ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state
tschettervictor commented 3 weeks ago

Any update? Bastille I unusable at this point with a 14.1 jail.