Support for a FreeBSD as a client OS

[yaroslav-gwit]

Is your feature request related to a problem? Please describe. First of all - thank you for this beautiful project and all your hard work! I wanted to introduce NetBird as a way to allow remote users access our FreeBSD virtualisation nodes.

FreeBSD bhyve VMs and FreeBSD Jails are using the private bridges (bridges that are not connected to any external interface) as internal networks and then the host node uses PF to orchestrate all the network traffic (NAT, VM-to-VM access, etc).

It's an SDN-like solution, which leverages site-to-site WG links for host-to-host communications (and all the internal route management), but it's very tedious to use the "naked" WG for every remote user (or client device) that wants to access our cluster(s).

We are using ZeroTier as of now to onboard the remote users/client devices, but I am not happy with their business licencing terms (I am a paid customer, using their Pro plan), as they are trying to convert all the Pro users to the Custom Enterprise Plan - Link to the Reddit discussion for those interested. I am also not happy with how ZT works under FreeBSD, but that's a whole another story on it's own.

Describe the solution you'd like The NetBird installation script didn't work for the FreeBSD (nor that I expected it to, because the OS is not on the "supported" list, but it doesn't hurt to try), and I could not find any build instructions to compile the client manually (could be just me skimming thought the repo and missing the build/make script).

Ideally, I'd like to see the NetBird deployment script include an automatic deployment options for FreeBSD, but it will likely take a long time to implement. For now I tried to compile the client like so (but the build has failed):

git clone ${NetBird_Repo}
cd netbird/client
go121 build

Here is output:

# github.com/netbirdio/netbird/client/ssh
ssh/server.go:193:5: undefined: setWinSize
# github.com/netbirdio/netbird/client/internal/dns
internal/dns/server.go:149:25: s.initialize undefined (type *DefaultServer has no field or method initialize, but does have Initialize)

Describe alternatives you've considered As an alternative, I tried using a Linux VM running under the FreeBSD to deploy the service and distribute the routes. It works well, but far from ideal - it's harder to automate PF around this process, and we lose the autostart if the VM is running on top of the encrypted ZFS dataset (it becomes a manual process to unlock the dataset first, and then start the VM).

Additional context I am happy to provide any additional context on request.

[skillcoder]

Seems setWinSize can be ignored on headless freebsd with a new file ssh/window_freebsd.go with following content

//go:build freebsd

package ssh

import (
    "os"
)

func setWinSize(file *os.File, width, height int) {
}

But regarding network implementation need to research. I will try to reuse already implemented approach but with freebsd build tag After re-implement dns management need to implement these for freebsd

internal/config.go:33:48: undefined: iface.WgInterfaceDefault
internal/config.go:188:25: undefined: iface.WgInterfaceDefault
internal/engine.go:1164:15: undefined: iface.NewWGIFace
internal/engine.go:1170:23: e.wgInterface.CreateOnAndroid undefined (type *iface.WGIface has no field or method CreateOnAndroid)

and much more in netbird/iface and

cmd/ssh.go:55:12: undefined: util.IsAdmin

[skillcoder]

I'm able to compile client, but I'm afraid current implementation know nothing about proper rc.d script and etc location in freebsd.

# ./netbird login
Error: failed to connect to daemon error: context deadline exceeded
If the daemon is not running please run: 
netbird service install 
netbird service start

@yaroslav-gwit Could you please share your rc script and how you setup settings for initial run, for example NB_SETUP_KEY I will try to proper implement netbird up command As I understand need to create rc script in /usr/local/etc/rc.d/ and generate config.json in /usr/local/etc/netbird/ Probably also we will have problems with tun interface and wg kernel module, let's assume we install kernel module manually via sudo pkg install wireguard

[skillcoder]

I'm able to successfully registered in management server with following command

./netbird -c ./config.json up -F

2024-02-24T19:16:58+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service

but can't connect

2024-02-24T19:16:58+03:00 INFO client/internal/connect.go:96: starting NetBird client version development
2024-02-24T19:16:59+03:00 ERRO client/internal/engine.go:267: failed creating tunnel interface wt0: [not implemented]
2024-02-24T19:16:59+03:00 ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: not implemented
2024-02-24T19:17:01+03:00 ERRO client/internal/engine.go:267: failed creating tunnel interface wt0: [not implemented]
2024-02-24T19:17:01+03:00 ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: not implemented

[yaroslav-gwit]

Great news, thanks. I'll give it a go on my side, too.

Which WG implementation are you guys using under the hood? Did you integrate wireguard-go, or do you rely on the kernel code to provide WG support?

Both options work fine under FreeBSD, and WireGuard is even integrated into a default kernel (since 13.1, I think). To create a new WG interface, it's enough to execute:

ifconfig wg create

Example output:

wg0

Then you could simply do this to turn it into wt interface:

ifconfig wg0 name wt0

[skillcoder]

Thank you, @yaroslav-gwit. I will try to implement this approach. If you have more ideas on how it should work, please share them. Additionally, if you have a good rc script for Netbird, it would also be valuable to create a FreeBSD port to simplify Netbird installation on FreeBSD.

Which WG implementation are you guys using under the hood? Did you integrate wireguard-go, or do you rely on the kernel code to provide WG support?

I'm currently know nothing about netbird, just start using it. @braginini Can anyone from the core team answer these questions? My current approach is to use wireguard package capabilities, just to make it work and later implement it in a most freebsd idiomatic way, I suppose ifconfig wg is best way. @yaroslav-gwit Please correct me if I'm wrong.

[skillcoder]

Meanwhile, I'm able to connect to the service on private node through netbird by using netstack mode

# on freebsd 13.0
export NB_USE_NETSTACK_MODE=true
export NB_SOCKS5_LISTENER_PORT=30000
./client -c ~/.netbird-config.json up -F
curl --socks5-hostname localhost:30000 100.12X.XXX.XXX:33000

[skillcoder]

The current problem is github.com/vishvananda/netlink not supported FreeBSD. The [not implemented] from this package. So need to make a PR to the netlink to add support for freebsd kernel networks interfaces, or for now create own wrapper for cli commands like ifconfig wg create

[skillcoder]

netlink protocol supported in FreeBSD since 13.2 But github.com/vishvananda/netlink looks very dead. Fortunately seems currently golang is implementing netlink support for freebsd in https://pkg.go.dev/golang.org/x/sys/unix https://groups.google.com/g/golang-codereviews/c/0YRr1-YEatI?pli=1

[yaroslav-gwit]

Re rc script: We could "borrow" some ideas from Nebula and Tailscale services. They both work really well under FreeBSD. I can look them up tomorrow at some point and post the results here.

Re netlink: Shell wrappers could be a good starting point before we have a proper kernel interface implementation. FreeBSD is not Linux, and ifconfig isn't going anywhere any time soon.

[yaroslav-gwit]

Sorry for the delay, here are the rc files (I will split it into 2 messages, so it's easier to read):

TailScale:

#!/bin/sh

# PROVIDE: tailscaled
# REQUIRE: NETWORKING
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# tailscaled_enable (bool):     Set it to YES to enable tailscaled.
#                               Default is "NO".
# tailscaled_state_dir (str):   Set the path to use for the state directory.
#                               Default is "/var/db/tailscale"
# tailscaled_port (number):     Set the port to listen on for incoming VPN packets.
#                               Default is "41641".
# tailscaled_syslog_output_enable (bool):       Set to enable syslog output.
#                                               Default is "NO". See daemon(8).
# tailscaled_syslog_output_priority (str):      Set syslog priority if syslog enabled.
#                                               Default is "info". See daemon(8).
# tailscaled_syslog_output_facility (str):      Set syslog facility if syslog enabled.
#                                               Default is "daemon". See daemon(8).
# tailscaled_exitnode_enable (bool):    Set it to YES to announce tailscaled as
#                                       an exit node. Default is "NO".
# tailscaled_up_args (str):             Additional arguments to pass to tailscale up
#                                       Default is "" (empty string).
# tailscaled_tun_dev (str):     Set the name of the tun interface tailscaled creates.
#                               Default is "tailscale0"

. /etc/rc.subr

name=tailscaled
rcvar=tailscaled_enable

load_rc_config $name

: ${tailscaled_enable:="NO"}
: ${tailscaled_state_dir:="/var/db/tailscale"}
: ${tailscaled_port:="41641"}
: ${tailscaled_exitnode_enable:="NO"}
: ${tailscaled_up_args:=""}
: ${tailscaled_tun_dev:="tailscale0"}

DAEMON=$(/usr/sbin/daemon 2>&1 | grep -q syslog ; echo $?)
if [ ${DAEMON} -eq 0 ]; then
        : ${tailscaled_syslog_output_enable:="NO"}
        : ${tailscaled_syslog_output_priority:="info"}
        : ${tailscaled_syslog_output_facility:="daemon"}
        if checkyesno tailscaled_syslog_output_enable; then
                tailscaled_syslog_output_flags="-t ${name} -T ${name}"

                if [ -n "${tailscaled_syslog_output_priority}" ]; then
                        tailscaled_syslog_output_flags="${tailscaled_syslog_output_flags} -s ${tailscaled_syslog_output_priority}"
                fi

                if [ -n "${tailscaled_syslog_output_facility}" ]; then
                        tailscaled_syslog_output_flags="${tailscaled_syslog_output_flags} -l ${tailscaled_syslog_output_facility}"
                fi
        fi
else
        tailscaled_syslog_output_enable="NO"
        tailscaled_syslog_output_flags=""
fi

pidfile=/var/run/${name}.pid
procname="/usr/local/bin/${name}"
ctlname="/usr/local/bin/tailscale"

start_cmd="${name}_start"
start_postcmd="${name}_poststart"
stop_postcmd="${name}_poststop"

tailscaled_start()
{
        # Check for orphaned tailscale network interface
        # And if it exists, then destroy it
        /sbin/ifconfig ${tailscaled_tun_dev} >/dev/null 2>&1 && (
                /sbin/ifconfig ${tailscaled_tun_dev} | fgrep -qw PID ||
                /sbin/ifconfig ${tailscaled_tun_dev} destroy
        )

        /usr/sbin/daemon -f ${tailscaled_syslog_output_flags} -p ${pidfile} ${procname} -port ${tailscaled_port} -tun ${tailscaled_tun_dev} -statedir ${tailscaled_state_dir}
}

tailscaled_poststart()
{
        if checkyesno tailscaled_exitnode_enable; then
                logger -s -t tailscale "Enabling Exit node mode"
                tailscaled_up_args=" --advertise-exit-node ${tailscaled_up_args}"
        fi
        if [ -n "${tailscaled_up_args}" ]; then
                ${ctlname} up ${tailscaled_up_args}
        fi
}

tailscaled_poststop()
{
        /sbin/ifconfig ${tailscaled_tun_dev} >/dev/null 2>&1 && (
                logger -s -t tailscaled "Destroying ${tailscaled_tun_dev} adapter"
                /sbin/ifconfig ${tailscaled_tun_dev} destroy || logger -s -t tailscaled "Failed to destroy ${tailscaled_tun_dev} adapter"
        )
}

run_rc_command "$1"

[yaroslav-gwit]

Nebula rc file:

#!/bin/sh 

# PROVIDE: nebula
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

# To enable nebula, add 'nebula_enable="YES"' to /etc/rc.conf or
# /etc/rc.conf.local

# Optional settings:
# nebula_config (string):   Full path to nebula configuration file
#                           (/usr/local/etc/nebula/config.yml)
# nebula_logfile (string):   Full path to nebula log file
#                           (/var/log/nebula.log)

. /etc/rc.subr 

name="nebula"
rcvar="nebula_enable"
desc="Scalable overlay networking tool with a focus on performance, simplicity and security"
pidfile="/var/run/nebula.pid"
procname="/usr/local/bin/nebula"

load_rc_config $name 

# Defaults
: ${nebula_enable:=NO}
: ${nebula_config:=/usr/local/etc/nebula/config.yml}
: ${nebula_logfile:=/var/log/nebula.log}

command="/usr/sbin/daemon"
command_args="-c -p ${pidfile} -t nebula -o ${nebula_logfile} ${procname} -config ${nebula_config}"

required_files="${nebula_config} ${command}"

extra_commands="configtest"
configtest_cmd="${procname} -test -config ${nebula_config}"

run_rc_command "$1"

[skillcoder]

Unfortunately root is required

ifconfig wg create
ifconfig: SIOCIFCREATE2: Operation not permitted

sudo ifconfig wg create
wg1

[yaroslav-gwit]

Yes, you do need root level privileges to execute most ifconfig commands that change network settings. And after all, it looks like both Nebula and Tailscale are running under root account in FreeBSD anyway.

Quick snapshot, of the process details for Tailscale on FreeBSD:

  PID USER       PRI  NI  VIRT   RES S  CPU%  MEM%   TIME+  Command
59733 root        52   0 1261M 32812 S   0.0  0.4  2:07.89  /usr/local/bin/tailscaled -port 41641 -tun tailscale0 -statedir /var/db/tailscale

[yaroslav-gwit]

A draft Go implementation could be something similar to this:

// This function takes in a new WT interface name, creates a new WG interface,
// and renames it to be `wt0` instead of the regular `wg0`.
//
// Returns an error, if there were any problems along the way.
func generateInterface(wtIfaceName string) error {
    // Get the effective user ID
    euid := os.Geteuid()
    // Check if the effective user ID is 0 (root)
    if euid != 0 {
        return fmt.Errorf("netbird must run as root on FreeBSD")
    }
    // Create a new WireGuard network interface
    out, err := exec.Command("ifconfig", "wg", "create").CombinedOutput()
    if err != nil {
        return fmt.Errorf("%s; %s", strings.TrimSpace(string(out)), err.Error())
    }
    // Rename the interface from WG to WT
    currentWgIface := strings.TrimSpace(string(out))
    out, err = exec.Command("ifconfig", currentWgIface, "name", wtIfaceName).CombinedOutput()
    if err != nil {
        return fmt.Errorf("%s; %s", strings.TrimSpace(string(out)), err.Error())
    }

    return nil
}

[skillcoder]

Finally wg interface created: wt0 with freebsd ifconfig wrapper

wt0: flags=8080<NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 100.XX.XX.XX netmask 0xffff0000
    groups: wg
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>

But ... new portion of linux specific things need to re-implemented for freebsd

2024-03-30T20:13:34Z ERRO client/internal/engine.go:274: failed creating firewall manager: not implemented for this OS: freebsd
2024-03-30T20:13:34Z ERRO client/internal/engine.go:287: failed to pull up wgInterface [wt0]: Not supported OS freebsd. SharedSocket is only supported on Linux
2024-03-30T20:13:34Z ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: Not supported OS freebsd. SharedSocket is only supported on Linux

[skillcoder]

It looks like implementing kernel space support is too challenging since freebsd do not supported by external libs. For example https://github.com/mdlayher/socket (SetBPF method not implemented for FreeBSD) Also highly likely we will have a problem with DNS So for now I will try to implement netbird for freebsd in userspace.

[skillcoder]

Done, netbird on FreeBSD in userspace mode works. Now I will refactor current code and prepare PR for the merge. This unblock us to create port and package of netbird on freebsd.

[yaroslav-gwit]

Did you use WireGuard-Go as an underlying implementation?

[skillcoder]

Yes, it is the same as on linux in userspace mode. https://github.com/netbirdio/netbird/blob/9c2dc05df1a735adc4ab8ec19fc6371c528817c2/go.mod#L175C39-L175C73

[drewhemm]

Really hoping for netbird integration with OPNsense, that would be a compelling combination.

[yaroslav-gwit]

@skillcoder Could you please publish a walk-through on how to run NetBird on FreeBSD (a build process, commands to start the agent, etc)? I bet people will on-board it, and will start using it in various places (pfSense, OPNSense, Jails, etc).

[skillcoder]

@yaroslav-gwit This is how I did it:

• Install FreeBSD 13.2 (I have to use this outdated version for now)

• install dependencies and build netbird client

  pkg install sudo git vim-tiny go zsh
  git clone https://github.com/netbirdio/netbird.git
  cd netbird/client
  go121 build

• since netbird client can be run only as root, add yourself to wheel and uncomment %wheel in

  vim  /usr/local/etc/sudoers

• to be able to use export run zsh

  zsh

• Connect to network in NETSTACK_MODE (for now I use only this, since I already use to it) (But user-land wireguard also supported and you could configure it by yourself, I just never did it for now on my production servers)

  # still in ~/netbird/client
  export NB_USE_NETSTACK_MODE=true
  export NB_SOCKS5_LISTENER_PORT=30000
  sudo ./client -c ~/.netbird-config.json up -F

• Follow https://login.netbird.io/authorize?... link to add this machine to your netbird network (required only once) in separate terminal on the same server

• Copy link from browser and call with curl (required only once)

  curl "http://localhost:53000/?code=..."

• Check connection to some address inside our netbird network

  curl --socks5-hostname localhost:30000 100.12X.XXX.XXX:33000

It works, I just checked it with fresh FreeBSD 13.2 installation.

[skillcoder]

Do we still need FreeBSD port and binary package? Or someone already did it?

[skillcoder]

I will take FreeBSD port creation.

[skillcoder]

Done. PoC port created and it works.

Now in my test environment setup process looks like this

cd /usr/ports/net/netbird
make install

# First start the netbird service:

sysrc netbird_enable=YES
service netbird start

# To connect to your netbird network:

netbird login

# NOTE: Use `curl "http://localhost:53000/?code=..."` on host which need to login if you setup netbird on remote host.

Before submitting this new port to the FreeBSD port committers, we need to resolve a few issues that the core team insists be addressed to ensure all functionality works as expected on FreeBSD and does not break anything. Specifically:

• Fix the skipped DNS management tests and add more tests.
• Fix the skipped Routing management tests and add more tests.

Additionally, I noticed that in my test environment, the Netbird client loses all connections every 30 seconds, which also needs to be fixed.

We also need to decide where to place the Netbird client config.json. I believe /etc/netbird is a very inappropriate location for FreeBSD, and /usr/local/etc/netbird/config.json is also not the best choice since this is an automatically generated config that we should never need to modify by hand. Therefore, I propose placing it in /var/db/netbird/, which is the standard location for system service state on FreeBSD.

Logs will be written to the standard location /var/log/netbird/client.log.

I want to enable --anonymize flag by default to simplify log sharing.

And will work on rc flags to be able easily configure RC script with standard FreeBSD way via rc.conf to pass custom:

• --setup-key
• --preshared-key
• --management-url
• --admin-url
• --config
• --log-level
• --daemon-addr
• --hostname
• disabling default --anonymize option

[skillcoder]

For config location I have created a separate issue https://github.com/netbirdio/netbird/issues/2383