Jomy10 commented 2 weeks ago

Describe the bug I can't create a pot using flavours. Creating a pot normally does work.

To Reproduce Consider these files:

test

set-attr -A no-rc-script -V YES
set-attr -A persisten -V NO
set-rss -C 1

test-cmd
```
set-cmd -c "python -m http.server"
```
test.sh
```
#!/bin/sh
```

[ -w /etc/pkg/FreeBSD.conf ] && sed -i '' 's/quarterly/latest/' /etc/pkg/FreeBSD.conf ASSUME_ALWAYS_YES=yes pkg bootstrap touch /etc/rc.conf sysrc sendmail_enable="NONE" pkg install -y python pkg clean -y


Now running `pot create -p test1 -b 14.1 -N public-bridge -t single -f test -f test-cmd` will result in the following error message:

Mon Sep 23 15:59:40 UTC 2024 test.sh -> /opt/pot/jails/test1/m/tmp/test.sh Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait... pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest. Consider changing PACKAGESITE. sendmail_enable: NONE -> NONE Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait... pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest. Consider changing PACKAGESITE. Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait... pkg: Error fetching https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest/Latest/pkg.txz: Host does not resolve Address resolution failed for https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest. Consider changing PACKAGESITE.

> create: flavour test failed (script)


**Expected behavior**
Not expecting any errors.

**System configuration - if possible**
 - `/usr/local/etc/pot/pot.conf`

pot configuration file

All datasets related to pot use the some zfs dataset as parent

With this variable, you can choose which dataset has to be used

POT_ZFS_ROOT=sys/pot

It is also important to know where the root dataset is mounted

POT_FS_ROOT=/opt/pot

This is the cache used to import/export pots

POT_CACHE=/var/cache/pot

This is where pot is going to store temporary files

POT_TMP=/tmp

This is the group owning POT_FS_ROOT

POT_GROUP=pot

This is the suffix added to temporary files created using mktemp,

X is a placeholder for a random character, see mktemp(1)

POT_MKTEMP_SUFFIX=.XXXXXXXX

Define the max length of the hostname inside the pot

POT_HOSTNAME_MAX_LENGTH=64

Internal Virtual Network configuration

IPv4 Internal Virtual network

POT_NETWORK=10.192.0.0/10

Internal Virtual Network netmask

POT_NETMASK=255.192.0.0

The default gateway of the Internal Virtual Network

POT_GATEWAY=10.192.0.1

The name of the network physical interface, to be used as default gateway

POT_EXTIF=vtnet0

The list of extra network interface, to make other network segments accessible

POT_EXTRA_EXTIF=vlan20 vlan50

for each extra interface, a variable is used to sepcify its network segment

POT_NETWORK_vlan20=192.168.100.0/24

POT_NETWORK_vlan50=10.50.50.0/24

Do not allow bridge-based pots to forward traffic to each other

POT_ISOLATE_VNET_POTS=true

DNS on the Internal Virtual Network

name of the pot running the DNS

POT_DNS_NAME=dns

IP of the DNS

POT_DNS_IP=10.192.0.2

VPN support

name of the tunnel network interface

POT_VPN_EXTIF=tun0

POT_VPN_NETWORKS=192.168.0.0/24 192.168.10.0/24


- System

` root@xxxx.xx ` `.....---.......--. -/ ------------- +o .--/y: +. OS: FreeBSD 14.1-RELEASE amd64 yo:. :o+- Uptime: 5 days, 21 hours, 39 mins y/ -/-o/ Packages: 128 (pkg) .- ::/sy+:. Shell: zsh 5.9 /-- / Terminal: /dev/pts/0 : : CPU: Intel Xeon (Skylake, IBRS, no TSX) (2) @ 2.294GHz : : GPU: Virtio 1.0 GPU / / Memory: 3571MiB / 3957MiB .- -. -- -. : : .-- `--. .---.....----.


This is an instance running on a Hetzner VPS.

** If network related **
 - `cat /etc/pf.conf`

Firewall

/etc/pf.conf

vim: set ft=pf

##########

Macros

##########

PODMAN

Change these to the interface(s) with the default route

v4egress_if = "vtnet0" v6egress_if = "vtnet0"

v4egress_if = "ix0"

v6egress_if = "ix0"

END PODMAN

Set public interface

ext_if = "vtnet0"

Set server public IP address

ext_if_ip = ""

Set and drop IP ranges on public interface

martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \ 10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \ 0.0.0.0/8, 240.0.0.0/4 }"

Set http (80) and https (443) ports

webports = "{http, https}"

enable services

int_tcp_services = "{domain, ntp, smtp, www, https, ftp, ssh}" # mail also goes here int_udp_services = "{domain, ntp}"

##########

Tables

##########

PODMAN

table

END PODMAND

###########

Options

###########

Skip loop back interface

set skip on lo

Sets the interface for which PF should gather statistics such as bytes in/out and packets passed/blocked

set loginterface $ext_if

######################

Ethernet filtering

######################

#########################

Traffic normalization

#########################

Deal with attacks based on incorrect handling of packet fragments

scrub in all

############

Queueing

############

###############

Translation

###############

PODMAN

nat on $v4egress_if inet from to any -> ($v4egress_if) nat on $v6egress_if inet6 from to !ff00::/8 -> ($v6egress_if)

rdr-anchor "cni-rdr/" nat-anchor "cni-rdr/"

END PODMAN

POT

nat-anchor pot-nat rdr-anchor "pot-rdr/*"

END POT

####################

Packet Filtering

####################

action [direction] [log] [quick] [on interface] [af] [proto protocol]

[from src_addr [port src_port]] [to dst_addr [port dst_port]]

[flags tcp_flags] [state]

Set default policy

block return in log all block out all

Drop all Non-Routable Addresses

block drop in quick on $ext_if from $martians to any block drop out quick on $ext_if from any to $martians

Blocking spoofed packets

antispoof quick for $ext_if

Allow SSH from any IP address

pass in inet proto tcp to $ext_if port ssh

allow tcp and udp traffic from 10.88.0.0/16

pass in inet proto tcp from 10.88.0.0/16 to $ext_if pass in inet proto udp from 10.88.0.0/16 to $ext_if

Allow Ping-Pong stuff. Be a good sysadmin

pass inet proto icmp icmp-type echoreq

All access to our Nginx/Apache/Lighttpd Webserver ports

pass proto tcp from any to $ext_if port $webports

Allow essential outgoing traffic

pass out quick on $ext_if proto tcp to any port $int_tcp_services pass out quick on $ext_if proto udp to any port $int_udp_services

Podman dns

pass in on cni-podman0

 - `potnet show -v`

16:09:46 [ INFO] Insert network 10.192.0.0/10 16:09:46 [ INFO] Insert broadcast 10.192.0.0/10 16:09:46 [ INFO] Insert gateway 10.192.0.1 16:09:46 [ INFO] Insert dns 10.192.0.2 Network topology: network : 10.192.0.0/10 min addr: 10.192.0.0 max addr: 10.255.255.255

Addresses already taken: 10.192.0.0 10.192.0.1 default gateway 10.192.0.2 dns 10.255.255.255

Debug information PotSystemConfig { zfs_root: "sys/pot", fs_root: "/opt/pot", network: 10.192.0.0/10, netmask: 255.192.0.0, gateway: 10.192.0.1, ext_if: "vtnet0", dns: Some( PotDnsConfig { pot_name: "dns", ip: 10.192.0.2, }, ), }



**Additional context**
The system is running on a Hetzner VPS.

grembo commented 2 weeks ago

Hi,

What‘s in your jailhosts /etc/resolv.conf?

By default, pot inherits this setting from the jailhost. You can override it, see pot help create (or pot create help?).

I see that you have pot anchors in your pf.conf, but didn’t study it in detail if there might be a nat issue. So if the dns setting doesn’t solve it, place something like sleep 1000 in the flavor script and then jexec into the jail to debug nat (run something like host google.com 8.8.8.8 and seeing that works).

Jomy10 commented 2 weeks ago

@grembo /etc/resolv.conf is:

# Generated by resolvconf
nameserver 185.12.64.1
nameserver 185.12.64.2

The weird thing is that creating a pot without a flavour does work (for instance pot create -p test -b 14.1 -t single). When I go into the pot, I can install packages, so internet access does work.

I will see if I can debug any possible nat issues.

Jomy10 commented 2 weeks ago

I went into the jail as you suggested. I looked at the pf logs and saw a block on bridge1. So I added pass in on bridge1 to pf.conf. Now it works.

grembo commented 2 weeks ago

Based on your description it seems like no NAT rules are placed in the anchor while creating a flavor. This could be considered both a bug or a feature - it's certainly unexpected and should be addressed IMHO. Probably would make sense to have a flag to control behavior.

bsdpot / pot

[BUG] Address resolution failed while creating pot with flavours #299

> create: flavour test failed (script)

pot configuration file

All datasets related to pot use the some zfs dataset as parent

With this variable, you can choose which dataset has to be used

It is also important to know where the root dataset is mounted

This is the cache used to import/export pots

This is where pot is going to store temporary files

This is the group owning POT_FS_ROOT

POT_GROUP=pot

This is the suffix added to temporary files created using mktemp,

X is a placeholder for a random character, see mktemp(1)

POT_MKTEMP_SUFFIX=.XXXXXXXX

Define the max length of the hostname inside the pot

POT_HOSTNAME_MAX_LENGTH=64

Internal Virtual Network configuration

IPv4 Internal Virtual network

POT_NETWORK=10.192.0.0/10

Internal Virtual Network netmask

POT_NETMASK=255.192.0.0

The default gateway of the Internal Virtual Network

POT_GATEWAY=10.192.0.1

The name of the network physical interface, to be used as default gateway

The list of extra network interface, to make other network segments accessible

POT_EXTRA_EXTIF=vlan20 vlan50

for each extra interface, a variable is used to sepcify its network segment

POT_NETWORK_vlan20=192.168.100.0/24

POT_NETWORK_vlan50=10.50.50.0/24

Do not allow bridge-based pots to forward traffic to each other

POT_ISOLATE_VNET_POTS=true

DNS on the Internal Virtual Network

name of the pot running the DNS

POT_DNS_NAME=dns

IP of the DNS

POT_DNS_IP=10.192.0.2

VPN support

name of the tunnel network interface

POT_VPN_EXTIF=tun0

POT_VPN_NETWORKS=192.168.0.0/24 192.168.10.0/24

Firewall

/etc/pf.conf

vim: set ft=pf

Macros

PODMAN

Change these to the interface(s) with the default route

v4egress_if = "ix0"

v6egress_if = "ix0"

END PODMAN

Set public interface

Set server public IP address

Set and drop IP ranges on public interface

Set http (80) and https (443) ports

enable services

Tables

PODMAN

END PODMAND

Options

Skip loop back interface

Sets the interface for which PF should gather statistics such as bytes in/out and packets passed/blocked

Ethernet filtering

Traffic normalization

Deal with attacks based on incorrect handling of packet fragments

Queueing

Translation

PODMAN

END PODMAN

POT

END POT

Packet Filtering

action [direction] [log] [quick] [on interface] [af] [proto protocol]

[from src_addr [port src_port]] [to dst_addr [port dst_port]]

[flags tcp_flags] [state]

Set default policy

Drop all Non-Routable Addresses

Blocking spoofed packets

Allow SSH from any IP address

allow tcp and udp traffic from 10.88.0.0/16

Allow Ping-Pong stuff. Be a good sysadmin

All access to our Nginx/Apache/Lighttpd Webserver ports