Open yaroslav-gwit opened 7 months ago
Seems setWinSize can be ignored on headless freebsd with a new file ssh/window_freebsd.go
with following content
//go:build freebsd
package ssh
import (
"os"
)
func setWinSize(file *os.File, width, height int) {
}
But regarding network implementation need to research.
I will try to reuse already implemented approach but with freebsd
build tag
After re-implement dns management need to implement these for freebsd
internal/config.go:33:48: undefined: iface.WgInterfaceDefault
internal/config.go:188:25: undefined: iface.WgInterfaceDefault
internal/engine.go:1164:15: undefined: iface.NewWGIFace
internal/engine.go:1170:23: e.wgInterface.CreateOnAndroid undefined (type *iface.WGIface has no field or method CreateOnAndroid)
and much more in netbird/iface
and
cmd/ssh.go:55:12: undefined: util.IsAdmin
I'm able to compile client, but I'm afraid current implementation know nothing about proper rc.d script and etc location in freebsd.
# ./netbird login
Error: failed to connect to daemon error: context deadline exceeded
If the daemon is not running please run:
netbird service install
netbird service start
@yaroslav-gwit Could you please share your rc script and how you setup settings for initial run, for example NB_SETUP_KEY
I will try to proper implement netbird up
command
As I understand need to create rc script in /usr/local/etc/rc.d/
and generate config.json
in /usr/local/etc/netbird/
Probably also we will have problems with tun
interface and wg
kernel module, let's assume we install kernel module manually via sudo pkg install wireguard
I'm able to successfully registered in management server with following command
./netbird -c ./config.json up -F
2024-02-24T19:16:58+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
but can't connect
2024-02-24T19:16:58+03:00 INFO client/internal/connect.go:96: starting NetBird client version development
2024-02-24T19:16:59+03:00 ERRO client/internal/engine.go:267: failed creating tunnel interface wt0: [not implemented]
2024-02-24T19:16:59+03:00 ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: not implemented
2024-02-24T19:17:01+03:00 ERRO client/internal/engine.go:267: failed creating tunnel interface wt0: [not implemented]
2024-02-24T19:17:01+03:00 ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: not implemented
Great news, thanks. I'll give it a go on my side, too.
Which WG implementation are you guys using under the hood? Did you integrate wireguard-go
, or do you rely on the kernel code to provide WG support?
Both options work fine under FreeBSD, and WireGuard is even integrated into a default kernel (since 13.1, I think). To create a new WG interface, it's enough to execute:
ifconfig wg create
Example output:
wg0
Then you could simply do this to turn it into wt
interface:
ifconfig wg0 name wt0
Thank you, @yaroslav-gwit. I will try to implement this approach. If you have more ideas on how it should work, please share them. Additionally, if you have a good rc
script for Netbird, it would also be valuable to create a FreeBSD port to simplify Netbird installation on FreeBSD.
Which WG implementation are you guys using under the hood? Did you integrate wireguard-go, or do you rely on the kernel code to provide WG support?
I'm currently know nothing about netbird, just start using it.
@braginini Can anyone from the core team answer these questions?
My current approach is to use wireguard
package capabilities, just to make it work and later implement it in a most freebsd idiomatic way, I suppose ifconfig wg
is best way. @yaroslav-gwit Please correct me if I'm wrong.
Meanwhile, I'm able to connect to the service on private node through netbird by using netstack mode
# on freebsd 13.0
export NB_USE_NETSTACK_MODE=true
export NB_SOCKS5_LISTENER_PORT=30000
./client -c ~/.netbird-config.json up -F
curl --socks5-hostname localhost:30000 100.12X.XXX.XXX:33000
The current problem is github.com/vishvananda/netlink
not supported FreeBSD.
The [not implemented]
from this package.
So need to make a PR to the netlink
to add support for freebsd kernel networks interfaces, or for now create own wrapper for cli commands like ifconfig wg create
netlink
protocol supported in FreeBSD since 13.2
But github.com/vishvananda/netlink
looks very dead.
Fortunately seems currently golang is implementing netlink support for freebsd in https://pkg.go.dev/golang.org/x/sys/unix
https://groups.google.com/g/golang-codereviews/c/0YRr1-YEatI?pli=1
Re rc
script:
We could "borrow" some ideas from Nebula and Tailscale services. They both work really well under FreeBSD. I can look them up tomorrow at some point and post the results here.
Re netlink
:
Shell wrappers could be a good starting point before we have a proper kernel interface implementation. FreeBSD is not Linux, and ifconfig
isn't going anywhere any time soon.
Sorry for the delay, here are the rc
files (I will split it into 2 messages, so it's easier to read):
TailScale:
#!/bin/sh
# PROVIDE: tailscaled
# REQUIRE: NETWORKING
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# tailscaled_enable (bool): Set it to YES to enable tailscaled.
# Default is "NO".
# tailscaled_state_dir (str): Set the path to use for the state directory.
# Default is "/var/db/tailscale"
# tailscaled_port (number): Set the port to listen on for incoming VPN packets.
# Default is "41641".
# tailscaled_syslog_output_enable (bool): Set to enable syslog output.
# Default is "NO". See daemon(8).
# tailscaled_syslog_output_priority (str): Set syslog priority if syslog enabled.
# Default is "info". See daemon(8).
# tailscaled_syslog_output_facility (str): Set syslog facility if syslog enabled.
# Default is "daemon". See daemon(8).
# tailscaled_exitnode_enable (bool): Set it to YES to announce tailscaled as
# an exit node. Default is "NO".
# tailscaled_up_args (str): Additional arguments to pass to tailscale up
# Default is "" (empty string).
# tailscaled_tun_dev (str): Set the name of the tun interface tailscaled creates.
# Default is "tailscale0"
. /etc/rc.subr
name=tailscaled
rcvar=tailscaled_enable
load_rc_config $name
: ${tailscaled_enable:="NO"}
: ${tailscaled_state_dir:="/var/db/tailscale"}
: ${tailscaled_port:="41641"}
: ${tailscaled_exitnode_enable:="NO"}
: ${tailscaled_up_args:=""}
: ${tailscaled_tun_dev:="tailscale0"}
DAEMON=$(/usr/sbin/daemon 2>&1 | grep -q syslog ; echo $?)
if [ ${DAEMON} -eq 0 ]; then
: ${tailscaled_syslog_output_enable:="NO"}
: ${tailscaled_syslog_output_priority:="info"}
: ${tailscaled_syslog_output_facility:="daemon"}
if checkyesno tailscaled_syslog_output_enable; then
tailscaled_syslog_output_flags="-t ${name} -T ${name}"
if [ -n "${tailscaled_syslog_output_priority}" ]; then
tailscaled_syslog_output_flags="${tailscaled_syslog_output_flags} -s ${tailscaled_syslog_output_priority}"
fi
if [ -n "${tailscaled_syslog_output_facility}" ]; then
tailscaled_syslog_output_flags="${tailscaled_syslog_output_flags} -l ${tailscaled_syslog_output_facility}"
fi
fi
else
tailscaled_syslog_output_enable="NO"
tailscaled_syslog_output_flags=""
fi
pidfile=/var/run/${name}.pid
procname="/usr/local/bin/${name}"
ctlname="/usr/local/bin/tailscale"
start_cmd="${name}_start"
start_postcmd="${name}_poststart"
stop_postcmd="${name}_poststop"
tailscaled_start()
{
# Check for orphaned tailscale network interface
# And if it exists, then destroy it
/sbin/ifconfig ${tailscaled_tun_dev} >/dev/null 2>&1 && (
/sbin/ifconfig ${tailscaled_tun_dev} | fgrep -qw PID ||
/sbin/ifconfig ${tailscaled_tun_dev} destroy
)
/usr/sbin/daemon -f ${tailscaled_syslog_output_flags} -p ${pidfile} ${procname} -port ${tailscaled_port} -tun ${tailscaled_tun_dev} -statedir ${tailscaled_state_dir}
}
tailscaled_poststart()
{
if checkyesno tailscaled_exitnode_enable; then
logger -s -t tailscale "Enabling Exit node mode"
tailscaled_up_args=" --advertise-exit-node ${tailscaled_up_args}"
fi
if [ -n "${tailscaled_up_args}" ]; then
${ctlname} up ${tailscaled_up_args}
fi
}
tailscaled_poststop()
{
/sbin/ifconfig ${tailscaled_tun_dev} >/dev/null 2>&1 && (
logger -s -t tailscaled "Destroying ${tailscaled_tun_dev} adapter"
/sbin/ifconfig ${tailscaled_tun_dev} destroy || logger -s -t tailscaled "Failed to destroy ${tailscaled_tun_dev} adapter"
)
}
run_rc_command "$1"
Nebula rc
file:
#!/bin/sh
# PROVIDE: nebula
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown
# To enable nebula, add 'nebula_enable="YES"' to /etc/rc.conf or
# /etc/rc.conf.local
# Optional settings:
# nebula_config (string): Full path to nebula configuration file
# (/usr/local/etc/nebula/config.yml)
# nebula_logfile (string): Full path to nebula log file
# (/var/log/nebula.log)
. /etc/rc.subr
name="nebula"
rcvar="nebula_enable"
desc="Scalable overlay networking tool with a focus on performance, simplicity and security"
pidfile="/var/run/nebula.pid"
procname="/usr/local/bin/nebula"
load_rc_config $name
# Defaults
: ${nebula_enable:=NO}
: ${nebula_config:=/usr/local/etc/nebula/config.yml}
: ${nebula_logfile:=/var/log/nebula.log}
command="/usr/sbin/daemon"
command_args="-c -p ${pidfile} -t nebula -o ${nebula_logfile} ${procname} -config ${nebula_config}"
required_files="${nebula_config} ${command}"
extra_commands="configtest"
configtest_cmd="${procname} -test -config ${nebula_config}"
run_rc_command "$1"
Unfortunately root is required
ifconfig wg create
ifconfig: SIOCIFCREATE2: Operation not permitted
sudo ifconfig wg create
wg1
Yes, you do need root level privileges to execute most ifconfig
commands that change network settings. And after all, it looks like both Nebula
and Tailscale
are running under root
account in FreeBSD anyway.
Quick snapshot, of the process details for Tailscale
on FreeBSD:
PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command
59733 root 52 0 1261M 32812 S 0.0 0.4 2:07.89 /usr/local/bin/tailscaled -port 41641 -tun tailscale0 -statedir /var/db/tailscale
A draft Go implementation could be something similar to this:
// This function takes in a new WT interface name, creates a new WG interface,
// and renames it to be `wt0` instead of the regular `wg0`.
//
// Returns an error, if there were any problems along the way.
func generateInterface(wtIfaceName string) error {
// Get the effective user ID
euid := os.Geteuid()
// Check if the effective user ID is 0 (root)
if euid != 0 {
return fmt.Errorf("netbird must run as root on FreeBSD")
}
// Create a new WireGuard network interface
out, err := exec.Command("ifconfig", "wg", "create").CombinedOutput()
if err != nil {
return fmt.Errorf("%s; %s", strings.TrimSpace(string(out)), err.Error())
}
// Rename the interface from WG to WT
currentWgIface := strings.TrimSpace(string(out))
out, err = exec.Command("ifconfig", currentWgIface, "name", wtIfaceName).CombinedOutput()
if err != nil {
return fmt.Errorf("%s; %s", strings.TrimSpace(string(out)), err.Error())
}
return nil
}
Finally wg interface created: wt0
with freebsd ifconfig
wrapper
wt0: flags=8080<NOARP,MULTICAST> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 100.XX.XX.XX netmask 0xffff0000
groups: wg
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
But ... new portion of linux specific things need to re-implemented for freebsd
2024-03-30T20:13:34Z ERRO client/internal/engine.go:274: failed creating firewall manager: not implemented for this OS: freebsd
2024-03-30T20:13:34Z ERRO client/internal/engine.go:287: failed to pull up wgInterface [wt0]: Not supported OS freebsd. SharedSocket is only supported on Linux
2024-03-30T20:13:34Z ERRO client/internal/connect.go:235: error while starting Netbird Connection Engine: Not supported OS freebsd. SharedSocket is only supported on Linux
It looks like implementing kernel space support is too challenging since freebsd do not supported by external libs. For example https://github.com/mdlayher/socket (SetBPF method not implemented for FreeBSD) Also highly likely we will have a problem with DNS So for now I will try to implement netbird for freebsd in userspace.
Done, netbird on FreeBSD in userspace mode works. Now I will refactor current code and prepare PR for the merge. This unblock us to create port and package of netbird on freebsd.
Did you use WireGuard-Go
as an underlying implementation?
Yes, it is the same as on linux in userspace mode. https://github.com/netbirdio/netbird/blob/9c2dc05df1a735adc4ab8ec19fc6371c528817c2/go.mod#L175C39-L175C73
Really hoping for netbird integration with OPNsense, that would be a compelling combination.
@skillcoder Could you please publish a walk-through on how to run NetBird on FreeBSD (a build process, commands to start the agent, etc)? I bet people will on-board it, and will start using it in various places (pfSense, OPNSense, Jails, etc).
@yaroslav-gwit This is how I did it:
Install FreeBSD 13.2 (I have to use this outdated version for now)
install dependencies and build netbird client
pkg install sudo git vim-tiny go zsh
git clone https://github.com/netbirdio/netbird.git
cd netbird/client
go121 build
since netbird client can be run only as root, add yourself to wheel and uncomment %wheel in
vim /usr/local/etc/sudoers
to be able to use export run zsh
zsh
Connect to network in NETSTACK_MODE (for now I use only this, since I already use to it) (But user-land wireguard also supported and you could configure it by yourself, I just never did it for now on my production servers)
# still in ~/netbird/client
export NB_USE_NETSTACK_MODE=true
export NB_SOCKS5_LISTENER_PORT=30000
sudo ./client -c ~/.netbird-config.json up -F
Follow https://login.netbird.io/authorize?... link to add this machine to your netbird network (required only once) in separate terminal on the same server
Copy link from browser and call with curl (required only once)
curl "http://localhost:53000/?code=..."
Check connection to some address inside our netbird network
curl --socks5-hostname localhost:30000 100.12X.XXX.XXX:33000
It works, I just checked it with fresh FreeBSD 13.2
installation.
Do we still need FreeBSD port and binary package? Or someone already did it?
I will take FreeBSD port creation.
Done. PoC port created and it works.
Now in my test environment setup process looks like this
cd /usr/ports/net/netbird
make install
# First start the netbird service:
sysrc netbird_enable=YES
service netbird start
# To connect to your netbird network:
netbird login
# NOTE: Use `curl "http://localhost:53000/?code=..."` on host which need to login if you setup netbird on remote host.
Before submitting this new port to the FreeBSD port committers, we need to resolve a few issues that the core team insists be addressed to ensure all functionality works as expected on FreeBSD and does not break anything. Specifically:
Additionally, I noticed that in my test environment, the Netbird client loses all connections every 30 seconds, which also needs to be fixed.
We also need to decide where to place the Netbird client config.json
. I believe /etc/netbird
is a very inappropriate location for FreeBSD, and /usr/local/etc/netbird/config.json
is also not the best choice since this is an automatically generated config that we should never need to modify by hand. Therefore, I propose placing it in /var/db/netbird/
, which is the standard location for system service state on FreeBSD.
Logs will be written to the standard location /var/log/netbird/client.log
.
I want to enable --anonymize
flag by default to simplify log sharing.
And will work on rc flags to be able easily configure RC script with standard FreeBSD way via rc.conf
to pass custom:
For config location I have created a separate issue https://github.com/netbirdio/netbird/issues/2383
Is your feature request related to a problem? Please describe. First of all - thank you for this beautiful project and all your hard work! I wanted to introduce
NetBird
as a way to allow remote users access our FreeBSD virtualisation nodes.FreeBSD
bhyve
VMs and FreeBSDJails
are using the private bridges (bridges that are not connected to any external interface) as internal networks and then the host node uses PF to orchestrate all the network traffic (NAT, VM-to-VM access, etc).It's an SDN-like solution, which leverages site-to-site WG links for host-to-host communications (and all the internal route management), but it's very tedious to use the "naked" WG for every remote user (or client device) that wants to access our cluster(s).
We are using ZeroTier as of now to onboard the remote users/client devices, but I am not happy with their business licencing terms (I am a paid customer, using their
Pro
plan), as they are trying to convert all thePro
users to theCustom Enterprise Plan
- Link to the Reddit discussion for those interested. I am also not happy with how ZT works under FreeBSD, but that's a whole another story on it's own.Describe the solution you'd like The
NetBird
installation script didn't work for the FreeBSD (nor that I expected it to, because the OS is not on the "supported" list, but it doesn't hurt to try), and I could not find any build instructions to compile the client manually (could be just me skimming thought the repo and missing the build/make script).Ideally, I'd like to see the
NetBird
deployment script include an automatic deployment options for FreeBSD, but it will likely take a long time to implement. For now I tried to compile the client like so (but the build has failed):Here is output:
Describe alternatives you've considered As an alternative, I tried using a Linux VM running under the FreeBSD to deploy the service and distribute the routes. It works well, but far from ideal - it's harder to automate PF around this process, and we lose the autostart if the VM is running on top of the encrypted ZFS dataset (it becomes a manual process to unlock the dataset first, and then start the VM).
Additional context I am happy to provide any additional context on request.