Use OpenBSD as NetVM

[tetrahedras]

The problem you're addressing (if any) The NetVM is exposed to a wide variety of attacks, and it's the first line of defense in hostile local network environments. (e.g there is evidence certain Middle Eastern countries have been building infrastructure to automatically attempt exploiting every device connected to public WLAN networks)

There is demand (see #4551) for running OpenBSD as a VM but efforts are stalled due to OpenBSD's lack of PV support. Since the NetVM is run in HVM mode, PV support is not necessary in this application.

Describe the solution you'd like OpenBSD offers well-reviewed driver code, high code quality, innovative exploit mitigations, excellent hardware support, low resource consumption, and a much smaller attack surface than Linux. OpenBSD also offers a much simpler approach to connecting to wireless networks than does Linux.

Where is the value to a user, and who might that user be? Adding it as a NetVM would likely increase resistance to attackers on the local network, and increase the diversity of the system (from the current-best MirageFW+Linux to OpenBSD+MirageFW+Linux).

Related, non-duplicate issues https://github.com/QubesOS/qubes-issues/issues/4551

[madaidan]

It is much better than Linux but OpenBSD isn't really the best choice. It doesn't have any meaningful security model for applications and instead just focuses on low level kernel stuff. The devs don't want to include any form of mandatory access control so there is no way to restrict a program's access except UNIX filesystem permissions and the OS is missing many mitigations such as CFI. It also doesn't really use any memory safe languages. The whonix wiki has some other criticisms here.

Instead, I'd recommend HardenedBSD.

Also see The insecurity of OpenBSD.

[esote]

@madaidan

It doesn't have any meaningful security model for applications

They introduced the pledge(2) and unveil(2) system calls specifically to secure user space. And also, improvements to kernel space imply improvements to user space. The security of latter relies on the former.

The devs don't want to include any form of mandatory access control so there is no way to restrict a program's access except UNIX file system permissions

See pledge(2). Mandatory access control and control-flow integrity are nice, but generally speaking they're messier, and if your system is more vulnerable through the lack of MAC and CFI then it's probably another issue.

MAC is less useful because Qubes OS is for single-user systems, and simply doing things in a disposable VM / separate VM is more useful than describing MAC policies for something.

However, like @tetrahedras describes, the security of attached network devices, etc. are more important and MAC would do little to help with this compared to using OpenBSD.

The whonix wiki has some other criticisms here.

The criticisms boil down to "OpenBSD has few users than Debian." Which is certainly true when compared to Debian, but compared to other BSDs bsdstats.org is not a reliable source for OpenBSD since it's opt-in (and almost no OpenBSD users would willingly send statistics), and Linus's law is not universal truth when most eyes aren't looking deeply enough.

Instead, I'd recommend HardenedBSD.

Unfortunatley it's only been around for 5-6 years, and is forked from FreeBSD which has a much worse record of security compared to OpenBSD (despite what HardenedBSD has done to remedy this).

[madaidan]

@esote

They introduced the pledge(2) and unveil(2) system calls specifically to secure user space.

They're only used if the application specifically uses it.

Compare that to a MAC system that enforces restrictions regardless if the application wants to use it or not and enforces restrictions on all apps by default.

See pledge(2).

Pledge just seems to restrict a few syscalls. The application still has access to the entire filesystem and other things.

Mandatory access control and control-flow integrity are nice, but generally speaking they're messier,

I don't see how they're any messier than pledge.

and if your system is more vulnerable through the lack of MAC and CFI then it's probably another issue.

That doesn't make sense.

"If your system is less secure due to not having security measures then it's probably not because it doesn't have security measures"

MAC is less useful because Qubes OS is for single-user systems, and simply doing things in a disposable VM / separate VM is more useful than describing MAC policies for something.

MAC will help with defense in depth. Why give the application even the opportunity to attempt to attack the hypervisor?

and is forked from FreeBSD which has a much worse record of security compared to OpenBSD

Criticisms against FreeBSD aren't criticisms against HardenedBSD. There are many differences between them.

[DemiMarie]

@esote

They introduced the pledge(2) and unveil(2) system calls specifically to secure user space.

They're only used if the application specifically uses it.

…which much of the OpenBSD base system does, as do ports of applications such as Chromium.

Compare that to a MAC system that enforces restrictions regardless if the application wants to use it or not and enforces restrictions on all apps by default.

In theory, yes. In practice, Android uses it to great effect, but desktop Linux distros virtually never do. Even on Fedora, users run as unconfined_t by default, which means that SELinux only restricts executable memory ― everything else is permitted. Even that may be off by default. Running as staff_t (the suggested default for users with administrative privileges) breaks basic things like user dbus instances.

Furthermore, Qubes VMs disable SELinux by default. With SELinux enforcing, they don’t boot without a significant Qubes-specific custom policy module. I have written such a module, but haven’t released it yet for want of time.

pledge(2) and unveil(2), on the other hand, are always on for the apps that use them ― and much of the OpenBSD base system does so.

See pledge(2).

Pledge just seems to restrict a few syscalls. The application still has access to the entire filesystem and other things.

It only has FS access if the pledge call says so. Many don’t.

[DemiMarie]

I am a massive supporter of this for multiple reasons.

Firstly, OpenBSD’s track record is legendary. Their slogan is “Only two remote holes in the default install, in a heck of a long time!”. The last such hole was in 2007 ― far before QubesOS even existed. Furthermore, my understanding (which could be wrong) is that “default install” encompasses all platforms that OpenBSD supports.

[madaidan]

which much of the OpenBSD base system does

That won't help restricting a newly installed application that contains malware or the many programs that don't use them such as Firefox.

In theory, yes. In practice, Android uses it to great effect, but desktop Linux distros virtually never do.

That doesn't mean Qubes shouldn't attempt it.

[esote]

which much of the OpenBSD base system does

That won't help restricting a newly installed application that contains malware or the many programs that don't use them such as Firefox.

On Qubes you should run untrusted, potentially-malicious programs in an untrusted VM, or in a disposable VM. The issue here is about establishing further security for things running with attached PCI devices like sys-net or sys-usb, since OpenBSD can work with HVM qubes.

In theory, yes. In practice, Android uses it to great effect, but desktop Linux distros virtually never do.

That doesn't mean Qubes shouldn't attempt it.

It means it's hard to get right, and very finicky. Due to its nature, Android gives users a much more restricted view of the underlying OS, compared to normal GNU/Linux desktops, and takes advantage of that.

[tetrahedras]

I've made some progress on getting OpenBSD to work as a NetVM, but am stuck at the point where other VMs have to connect to the new sys-net-bsd. Qubes doesn't seem to recognise that sys-net-bsd has booted, and so it won't let other VMs start up if they use sys-net-openbsd for networking.

Steps I took to get this far:

• create a HVM VM, we'll call it sys-net-bsd, and make sure to check the "provides network" box in settings
• leave it connected to networking for now
• download the installXX.iso install image from openbsd.org
• boot the VM with qvm-start sys-net-bsd --cdrom VM_WITH_INSTALL_IMAGE:/path/to/installXX.iso
• install
• set up gateway with route add default GATEWAY_IP
• create a hostname.if file vi /etc/hostname.xnf0 file contents: inet $VM_IP $VM_NETMASK NONE where NONE is a special keyword indicating we should derive broadcast from netmask
• re-run netstart sh /etc/netstart
• download all firmware (in case needed for wifi) fw_update -a
• shut down
• connect the wifi / network card to the OpenBSD VM
• start up openbsd
• connect to wifi ifconfig WIFI_INTERFACE join NETWORK_NAME wpakey NETWORK_PASSWORD

(and this is as far as I got, as Qubes doesn't seem to recognise the OpenBSD NetVM has finished booting and so I can't connect any downstream VMs to it)

Next steps would be:

• configure the internal interface with the same IP address as sys-net
• ensure we route from the internal interface to the wifi interface
• (optionally) turn off GUI terminal by default
  1. Set "debug" to "False" in qvm-prefs
  2. Set "gui" to "False" and "gui-emulated" to "False"

[esote]

@tetrahedras Thanks for sharing your steps! I'll try them on my machine this week.

Qubes doesn't seem to recognise that sys-net-bsd has booted

Have you tried setting the hostctl properties described in xen(4)?

[DemiMarie]

@esote

They introduced the pledge(2) and unveil(2) system calls specifically to secure user space.

They're only used if the application specifically uses it.

Compare that to a MAC system that enforces restrictions regardless if the application wants to use it or not and enforces restrictions on all apps by default.

See pledge(2).

Pledge just seems to restrict a few syscalls. The application still has access to the entire filesystem and other things.

Mandatory access control and control-flow integrity are nice, but generally speaking they're messier,

I don't see how they're any messier than pledge.

and if your system is more vulnerable through the lack of MAC and CFI then it's probably another issue.

That doesn't make sense.

"If your system is less secure due to not having security measures then it's probably not because it doesn't have security measures"

MAC is less useful because Qubes OS is for single-user systems, and simply doing things in a disposable VM / separate VM is more useful than describing MAC policies for something.

MAC will help with defense in depth. Why give the application even the opportunity to attempt to attack the hypervisor?

and is forked from FreeBSD which has a much worse record of security compared to OpenBSD

Criticisms against FreeBSD aren't criticisms against HardenedBSD. There are many differences between them.

The vast majority of the attack surface against a NetVM is the kernel-mode drivers. OpenBSD’s kernel is a far, far harder target than Linux. It has less code, and the mitigations that OpenBSD is famous for mean that exploiting the bugs that are there is hard.

[tetrahedras]

Qubes just posted a security bulletin which would potentially allow a compromised NetVM to fully compromise a Qubes install: https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-058-2020.txt

A hardened NetVM (e.g using OpenBSD) would make such an attack more difficult.

[DemiMarie]

tl;dr: Using OpenBSD as a NetVM will be hard. It lacks features found in most major *nix-like OSs, and workarounds may very well require kernel source code patching.

Getting a reasonable experience that was fail-safe required patching /etc/rc (!), which is not supported. I might be able to use a shell command run from /etc/hostname.xnf0, but that is an ugly hack.

OpenBSD also does not implement netback drivers, nor does it support hotplug of netfront interfaces. This means that the relationship between sys-net and sys-firewall would need to be changed. Furthermore, it does not seem to support specifying a route purely by network interface name ― one must always specify the address of that interface. That causes nasty problems for QubesOS, since every netback interface and the netfront interface of a qube all share an IP address. Link-local addresses are a workaround for IPv6, but I was not able to find a workaround for IPv4. There might be a workaround using pf (Packet Filter, the OpenBSD firewall), but I gave up before attempting one. There is a workaround: route -n add -inet "$IP" -iface -llinfo -static -link "$iface_name". This requires ARP to work, but a custom C program might be able to work around this limitation.

When it comes to filesystems, OpenBSD does not support bind mounts. I had to resort to a combination of loopback NFS mounts, symlinks, and ugly shell scripts to get the concept of TemplateVMs to work at all. Furthermore, I (again) had to patch /etc/rc to avoid deadlocks when shutting down from single-user mode, as the NFS server would go away before the mounts were unmounted. Finally, I wrote a C program that managed disklabels and handled creating the filesystems for AppVMs.

OpenBSD’s wireless hardware support is not the best. Even when a card is supported, it often is significantly slower than it is under Linux. This is not likely to change unless someone sponsors OpenBSD wireless driver development, and could very well be a dealbreaker for some people. There is also no support for NetworkManager, and (to my knowledge) no alternative GUI network management tools available.

Finally, and most importantly, OpenBSD’s Xen support is minimal and not well maintained. For several weeks, OpenBSD-CURRENT (the development branch) would panic on boot. It does not expose Xen hypercalls to userspace, which means that libvchan won’t work. Furthermore, OpenBSD does not have kernel modules, which means that adding such support requires building a custom kernel. Since most of how QubesOS integrates with its qubes is through vchans, this means that an OpenBSD HVM will always be a second-class citizen in QubesOS. Writing a driver that exposed the needed Xen features to userspace would be difficult, and upstreaming it even more difficult. Even if we did get the code upstream, we would need to maintain it indefinitely to prevent it from being removed later. I am certainly not willing to put in the effort required, so unless someone else is, good integration of QubesOS with OpenBSD won’t happen. And poor integration will be so user-unfriendly that I am not at all sure it would be worth the effort.

[DemiMarie]

One additional problem is that OpenBSD doesn’t support TRIM, so using it on thin-provisioned storage effectively leaks disk space. My OpenBSD-CURRENT qube quickly became one of the largest for this reason alone.

[brendanhoar]

Does the lvm thin back end “smartly” support zero-fill as discard with *bsd hvms? My Windows 7 hvms do, but that might be due to The Windows PV storage drivers. Worth experimenting?

[DemiMarie]

@brendanhoar Maybe. Have you considered upgrading to Windows 8.1 or 10? Windows 7 is EOL.

[brendanhoar]

Oh definitely, just hasn’t been a high priority as the win 7 VMs aren’t network connected.

Slowly working on my steps to set up some win 10 Lts and/windows server templates, but what with (tele-)work, full-time at-home parenting (yay! ...but overwhelmed) plus all the other stuff (gestures disapprovingly) going on...not quite there yet.

[tetrahedras]

On Wed, Sep 09, 2020 at 05:26:58PM -0700, Demi Marie Obenour wrote:

tl;dr: Using OpenBSD as a NetVM will be hard. It lacks features found in most major *nix-like OSs, and workarounds may very well require kernel source code patching.

Thanks for that very exhaustive writeup and all your work trying to get it running.

When it comes to filesystems, OpenBSD does not support bind mounts. I had to resort to a combination of loopback NFS mounts, symlinks, and ugly shell scripts to get the concept of TemplateVMs to work at all.
Furthermore, I (again) had to patch /etc/rc to avoid deadlocks when shutting down from single-user mode, as the NFS server would go away before the mounts were unmounted.

As an alternative to making an OpenBSD TemplateVM and Template-Based-VM, would it be easier to make a purely disposable NetVM?

There are already some people using a disp-sys-net (see the Qubes docs) so for someone who is paranoid enough to want an OpenBSD NetVM, making it disposable-only seems fair.

This would make it potentially possible to avoid bind mounts entirely, and simply clone the VM image on start. This is still sub-optimal, but probably better than patching /etc/rc.

[DemiMarie]

tl;dr: Using OpenBSD as a NetVM will be hard. It lacks features found in most major *nix-like OSs, and workarounds may very well require kernel source code patching.

Thanks for that very exhaustive writeup and all your work trying to get it running.

You're welcome! I am glad you found my write-up useful. Thanks for your time.

When it comes to filesystems, OpenBSD does not support bind mounts. I had to resort to a combination of loopback NFS mounts, symlinks, and ugly shell scripts to get the concept of TemplateVMs to work at all. Furthermore, I (again) had to patch /etc/rc to avoid deadlocks when shutting down from single-user mode, as the NFS server would go away before the mounts were unmounted.

As an alternative to making an OpenBSD TemplateVM and Template-Based-VM, would it be easier to make a purely disposable NetVM? There are already some people using a disp-sys-net (see the Qubes docs) so for someone who is paranoid enough to want an OpenBSD NetVM, making it disposable-only seems fair.

This would make it potentially possible to avoid bind mounts entirely, and simply clone the VM image on start. This is still sub-optimal, but probably better than patching /etc/rc.

That seems like a good option. Either a purely disposable or a purely persistent VM would be significantly easier than a TemplateBasedVM. I wound up patching /etc/rc in other ways as well, mostly to ensure that it dropped to a single-user shell in the event of errors.

We could avoid the loopback NFS mounts by replacing them with FUSE mounts, which OpenBSD does support. That still leaves the networking problem, though. Somehow, we would need to make sys-firewall be the backend for sys-net, not the frontend. And we would likely need to modify sys-net’s stubdomain to expose its console as an emulated serial port for sys-net, not as its own console. Without a serial console, submitting bug reports to upstream is a challenge, to put it mildly.

Finally, I believe that our chances of getting OpenBSD support in QubesOS would be vastly improved if we found an OpenBSD committer who was willing to help us.

[tetrahedras]

On Mon, Oct 12, 2020 at 11:27:50AM -0700, Demi Marie Obenour wrote:

That still leaves the networking problem, though. Somehow, we would need to make sys-firewall be the backend for sys-net, not the frontend.

Can you explain in more detail what you mean by this?

It sounds like you are suggesting that sys-firewall be the VM which talks to the NIC (or wireless LAN card). That would not make much sense, since the object of the exercise is to use OpenBSD's extreme hardening to protect against compromise via the NIC.

(we just saw a recent Xen vulnerability which would potentially allow someone to compromise dom0 by attacking the VM which talks to the NIC, and in particular wireless LAN driver vulnerabilities are fairly notorious.)

And we would likely need to modify sys-net’s stubdomain to expose its console as an emulated serial port for sys-net, not as its own console.

Maybe I misunderstood, but I thought xl console and qvm-console-dispvm (in dom0) used the serial console?

Without a serial console, submitting bug reports to upstream is a challenge, to put it mildly.

Finally, I believe that our chances of getting OpenBSD support in QubesOS would be vastly improved if we found an OpenBSD committer who was willing to help us.

Should we just don our asbestos suits and email the OpenBSD lists (with a link to this issue) to ask, or what would you suggest?

[unman]

On Tue, Oct 13, 2020 at 02:47:49AM -0700, tetrahedras wrote:

On Mon, Oct 12, 2020 at 11:27:50AM -0700, Demi Marie Obenour wrote:

That still leaves the networking problem, though. Somehow, we would need to make sys-firewall be the backend for sys-net, not the frontend.

Can you explain in more detail what you mean by this?

It sounds like you are suggesting that sys-firewall be the VM which talks to the NIC (or wireless LAN card). That would not make much sense, since the object of the exercise is to use OpenBSD's extreme hardening to protect against compromise via the NIC.

(we just saw a recent Xen vulnerability which would potentially allow someone to compromise dom0 by attacking the VM which talks to the NIC, and in particular wireless LAN driver vulnerabilities are fairly notorious.)

And we would likely need to modify sys-net???s stubdomain to expose its console as an emulated serial port for sys-net, not as its own console.

Maybe I misunderstood, but I thought xl console and qvm-console-dispvm (in dom0) used the serial console?

Without a serial console, submitting bug reports to upstream is a challenge, to put it mildly.

Finally, I believe that our chances of getting OpenBSD support in QubesOS would be vastly improved if we found an OpenBSD committer who was willing to help us.

Should we just don our asbestos suits and email the OpenBSD lists (with a link to this issue) to ask, or what would you suggest?

If you look in the more general openBSD issue - #4551 -, or search the mailing list, you'll see that I use an openBSD HVM in place of sys-net. It isn't difficult. It's just the same as using a Windows box as sys-net. It's almost trivial to have sys-firewall as backend for sys-net and requires absolutely minimal changes to routing and firewalling. Yes, it's an HVM and not template based, but if you want it (need it?), you can do this right now.

[marmarek]

If you look in the more general openBSD issue - #4551 -, or search the mailing list, you'll see that I use an openBSD HVM in place of sys-net. It isn't difficult. It's just the same as using a Windows box as sys-net. It's almost trivial to have sys-firewall as backend for sys-net and requires absolutely minimal changes to routing and firewalling. Yes, it's an HVM and not template based, but if you want it (need it?), you can do this right now.

@unman this response is not very helpful. It may be easy for you to do such change, but claiming you did this doesn't help others with actually doing it.

[DemiMarie]

That still leaves the networking problem, though. Somehow, we would need to make sys-firewall be the backend for sys-net, not the frontend.

Can you explain in more detail what you mean by this? It sounds like you are suggesting that sys-firewall be the VM which talks to the NIC (or wireless LAN card). That would not make much sense, since the object of the exercise is to use OpenBSD's extreme hardening to protect against compromise via the NIC. (we just saw a recent Xen vulnerability which would potentially allow someone to compromise dom0 by attacking the VM which talks to the NIC, and in particular wireless LAN driver vulnerabilities are fairly notorious.)

OpenBSD has a netfront driver (xnf(4)), but not a netback driver. So sys-net would have the NIC, but it would use the netfront driver to connect to sys-firewall, as opposed to the current situation where it uses the netback driver.

And we would likely need to modify sys-net’s stubdomain to expose its console as an emulated serial port for sys-net, not as its own console.

Maybe I misunderstood, but I thought xl console and qvm-console-dispvm (in dom0) used the serial console?

They use a paravirtualized console, which OpenBSD does not support. Getting PV console support into OpenBSD would be great, but would require someone to do the work.

Without a serial console, submitting bug reports to upstream is a challenge, to put it mildly. Finally, I believe that our chances of getting OpenBSD support in QubesOS would be vastly improved if we found an OpenBSD committer who was willing to help us.

Should we just don our asbestos suits and email the OpenBSD lists (with a link to this issue) to ask, or what would you suggest?

If you could do so, that would be great.

[unman]

On Tue, Oct 13, 2020 at 07:27:39AM -0700, Marek Marczykowski-G??recki wrote:

If you look in the more general openBSD issue - #4551 -, or search the mailing list, you'll see that I use an openBSD HVM in place of sys-net. It isn't difficult. It's just the same as using a Windows box as sys-net. It's almost trivial to have sys-firewall as backend for sys-net and requires absolutely minimal changes to routing and firewalling. Yes, it's an HVM and not template based, but if you want it (need it?), you can do this right now.

@unman this response is not very helpful. It may be easy for you to do such change, but claiming you did this doesn't help others with actually doing it.

No, but I'm working off my phone, and I have covered this a number of times on the mailing list, which is why I suggested searching there. Also, I'm pretty sure I wrote up some notes, again referred to on the mailing list.

[tetrahedras]

On Tue, Oct 13, 2020 at 09:05:04AM -0700, unman wrote:

@unman this response is not very helpful. It may be easy for you to do such change, but claiming you did this doesn't help others with actually doing it.

No, but I'm working off my phone, and I have covered this a number of times on the mailing list, which is why I suggested searching there. Also, I'm pretty sure I wrote up some notes, again referred to on the mailing list.

@unman thanks for writing up your experiences and telling us about it.

Here is unman's guide (from 2018): https://github.com/unman/notes/blob/master/openBSD_as_netvm

Here are the mailing list posts that I've been able to find:

Openbsd as a netvm https://groups.google.com/forum/#!searchin/qubes-users/openbsd$20unman|sort:date/qubes-users/MpXLhz5COvM/smHA6a0VBgAJ

Networking between qubes and HVMs https://groups.google.com/forum/#!searchin/qubes-users/openbsd$20unman|sort:date/qubes-users/4iU7UyNBj2A/O__FSl-pBAAJ

Using OpenBSD as Qubes firewall https://groups.google.com/forum/#!searchin/qubes-users/openbsd$20unman|sort:date/qubes-users/H7DRXS-Gds8/9141dAcECgAJ

Openbsd HVM - Network Issues https://groups.google.com/forum/#!searchin/qubes-users/openbsd$20unman|sort:date/qubes-users/qdSxRzI-Ejc/6JWYyEWDBgAJ

is it possible to have two sys-net for one firewall vm? https://groups.google.com/forum/#!searchin/qubes-users/openbsd$20sys-net|sort:date/qubes-users/SsTDITBDgBw/-XQs7-6rDAAJ

What you can do is this:

Net1-----sys-net1
               |
          sys-firewall
          |          |
Net2-----sys-net2   qube

sys-net2 has sys-firewall as netvm.
Attach NIC to sys-net2.

On sys-firewall you put custom rules that allow traffic between qube and
sys-net2.
You also need to set routing correctly,  modify raw table to allow
inbound traffic from Net2 on the sys-net2 vif.
If done right no configuration is needed on client qubes.
(You will, of course, need nat and filter rules on sys-net2 also.)

I do this to use openBSD HVMs as netVMs, and it works fine.

unman

[tetrahedras]

On Tue, Oct 13, 2020 at 08:13:31AM -0700, Demi Marie Obenour wrote:

If you could do so, that would be great.

I sent the following to misc@opensbsd.org but it has not yet appeared in the list archives so maybe it is being held for moderation, or maybe there is some kind of spam filter that ate it

Subject: Integrating OpenBSD into Xen/Qubes

A number of people are working on integrating OpenBSD into Qubes.

In particular, OpenBSD's hardening and mitigations are potentially very useful in talking to the NIC: Xen vulnerabilities have been repeatedly found that would allow a guest with PCI access to compromise the entire system, and on most machines the network card is a PCI device.

Additionally, wireless drivers on Linux leave some things to be desired and the network stack is very exposed to the adversary compared to other aspects of the system.

The limited scope of the networking VM in Qubes (it does not need much in the way of bells and whistles, it simply talks to the NIC and passes on data) means that it's much easier to use OpenBSD here than it would be to use OpenBSD for e.g GUI applications.

Unfortunately, there are still significant issues (currently good integration requires patching /etc/rc, among other things): https://github.com/QubesOS/qubes-issues/issues/5294#issuecomment-707278609

As the commenter notes, this would be much easier if an OpenBSD committer was interested in helping. Anyone?

[DemiMarie]

It has shown up @tetrahedras

[tetrahedras]

On Thu, Oct 15, 2020 at 07:37:38AM -0700, Demi Marie Obenour wrote:

It has shown up @tetrahedras

Thanks, I see it now.

[arkenoi]

..would also make sense for sys-firewall as well?

[DemiMarie]

@arkenoi Probably not. OpenBSD only offers a nefront driver (xnf(4)), not a netback driver.

[jcholsap]

UPDATED: I have an OpenBSD sys-net operational thanks to notes [1] and comments [2] from @unman. UPDATE_2: Since 2021 October I have refined the script and clarified the guide. It's been working flawlessly! Sorry, I'm not a programmer, just an integrator. So my GitHub skills are lacking. Any futher updates will be maintained here: https://github.com/jcholsap/freemod/issues/1#issue-1016495279

Many Qubes OS (Qubes) users could benefit from a full-featured, external facing firewall with capabilities such as IDPS. Since Qubes is a virtualization platform, it is reasonable to implement a hardened OS for this purpose.

This guide will get you an instance of OpenBSD that services the hardware Ethernet interface. You can then choose what functionality you want to install, such as a hardened router, a firewall, IDPS, etc.

Credit to notes [1] and comments [2] from @unman.

I highly recommend following these steps in the order presented. (See discussion at bottom for details.):

1. Preparations: a. Shutdown all VMs. b. Backup dom0, sys-net, and sys-firewall. c. All steps can be performed in the GUI except: -- i. Step 5, creating the openbsd-21 HVM TemplateVM, and -- ii. Step 10, running the script in sys-firewall.

2. In the sys-firewall VM change these settings: a. Confirm "Provides network" is set to "True". b. Set NetVM, aka "Networking", to none. c. Add in "Services" the following: -- i. "clocksync" daemon. -- ii. "qubes-update-check". -- iii. "qubes-updates-proxy". -- iv. "qubes-yum-proxy".

3. In the sys-net VM settings: a. Remove all hardware devices. b. Confirm "Provides network" is "True". c. Set NetVM, aka "Networking", to "sys-firewall". d. Change virtualization mode to "PVH". e. Change the following "Services": -- i. Remove "clock-sync". -- ii. Add "qubes-yum-proxy". -- iii. Add "qubes-updates-proxy". -- iv. Add "qubes-update-check".

4. Start the sys-firewall VM.

5. Create a HVM as a class of TemplateVM [3] [4] with these settings: a. Keeping the naming convention, my example uses the name "openbsd-21". b. dom0 command line (note that the kernel property is set to two single tick marks, aka, "none"): qvm-create openbsd-21 --class TemplateVM --property virt_mode=HVM --property kernel='' --label purple c. Set NetVM, aka "Networking", to "sys-firewall". d. Set "Start qube automatically on boot" to "False". e. Add network hardware "Devices". (For me it's 00:19.0 Ethernet controller: Intel Corp...) f. Confirm the kernel is set to "none".

6. Create a new HVM with class of AppVM using the openbsd-21 HVM TemplateVM in Step 5 above with these settings: a. Name "openbsd-21-sysnet". b. Set NetVM, aka "Networking", to "sys-firewall". c. Set "Start qube automatically on boot" to "False". d. Add network hardware "Devices". (For me it's 00:19.0 Ethernet controller: Intel Corp...)

7. Note the static IP assignment to "openbsd-21-sysnet" by Qubes. a. From Qubes Manager find the IP address assigned to the "openbsd-21-sysnet" VM. -- i. For me, it is 10.137.0.16 b. If you have any trouble with this step then try this: -- i. Reboot Qubes OS. -- ii. Start "openbsd-21-sysnet" VM. -- iii. Let it run until it at least identifies the Xen virtual ethernet interface "xn0". -- iv. Stop "openbsd-21-sysnet" VM.

8. Install OpenBSD in the "openbsd-21" HVM TemplateVM. a. Assign interfaces. Your LAN interface to sys-firewall is going to be xn? For me, LAN is xn0 and WAN is em0. b. Assign the static IP address from Step 7 above. For me it is 10.137.0.16 to the "xn0" interface. c. Change root password. d. Shutdown the "openbsd-21" TemplateVM.

9. Now start "openbsd-21-sysnet" and confirm the above settings copied from the parent "openbsd-21" TemplateVM: a. Interface assignments. (Again, my LAN is xn0 and WAN is em0). b. Interfaces' IP addresses. -- i. For me, WAN is set by DHCP and the LAN static is 10.137.0.16/24. -- ii. Note that this LAN static IP address was set by Qubes when openbsd-21-sysnet was instantiated in Step 7. c. For some reason I had to repeat the OpenBSD configuration in "openbsd-21" TemplateVM twice to make the root password change.

10. In "sys-firewall", create a script to route traffic to "openbsd-21-sysnet": a. The script needs to set the gateway, allow other VMs access, and identify the DNS server. For me it is like this:

    #!/usr/bin/sh
    #
    # Logging Function
    sysidps_log () { echo -e "$(date "+%F %T")" "$1" | tee -a >&1 "$log_file"; }
    log_msg_fail="Timed out waiting on gateway."
    log_msg_chk="Checking gateway for ICMP echo reply..."
    log_msg_up="Gateway replying to ping requests!"
    log_msg_rte="Gateway set as the default route."
    log_msg_gw="Virtual ifaces routing through gateway."
    log_msg_ns="Name servers added to routing table."
    log_file="/var/log/qubes/sys-idps"
    #
    # Hard code static IP addresses. You'll need at least one name server.
    gw_ip=10.137.0.14
    #ns1_ip=10.137.252.1
    ns2_ip=10.137.1.1
    #
    # Wait for gateway to come up.
    # Maximum wait time is 8 hrs before exits with failure.
    # Your Gateway Must Reply To (Does Not Block) ICMP Echo Requests!
    up_reply="0% packet loss,"
    ping_chk="false"
    declare -i time_out=0
    time_base=$(date "+%s")
    while [ "$ping_chk" != "$up_reply" ]; do
    if [ $time_out -lt 300 ] ; then sleep 1 ; 
    elif [ $time_out -lt 600 ] ; then sleep 3 ;
    elif [ $time_out -lt 1200 ] ; then sleep 5 ;
    elif [ $time_out -lt 1800 ] ; then sleep 8 ;
    elif [ $time_out -lt 3600 ] ; then sleep 15 ;
    elif [ $time_out -lt 14400 ] ; then sleep 30 ;
    elif [ $time_out -lt 28800 ] ; then sleep 60 ;
    else sysidps_log "$log_msg_fail" && exit 1 ;
    fi;
    sysidps_log "$log_msg_chk";
        ping_chk=$(ping -c 1 -r $gw_ip | grep "0% packet loss" | cut -d ' ' -f 6-8);
    time_out=$(date "+%s")-$time_base;
    done
    sysidps_log "$log_msg_up"
    #
    # Set gateway as default route.
    # Running "ip route add" first makes discovering gw_vif easy.
    ip route add default via $gw_ip
    gw_vif=$(ip route show default | cut -d ' ' -f 5)
    sysidps_log "$log_msg_rte"
    #
    # Add all VMs vif routing to/from gateway.
    iptables --insert FORWARD --in-interface vif+ --out-interface $gw_vif --jump ACCEPT
    iptables -t raw --insert PREROUTING --in-interface $gw_vif --jump ACCEPT
    sysidps_log "$log_msg_gw"
    #
    # Add routing to name server resolution. Call Qubes name server script.
    rm -f /etc/resolv.conf
    if [ "X$ns1_ip" != "X" ] ; then echo "nameserver $ns1_ip" >> /etc/resolv.conf ; fi
    if [ "X$ns2_ip" != "X" ] ; then echo "nameserver $ns2_ip" >> /etc/resolv.conf ; fi
    /usr/lib/qubes/qubes-setup-dnat-to-ns
    sysidps_log "$log_msg_ns"
    #
    exit 0

11. You could make this a background process in /rw/config/rc.local. If so, then set openbsd-21-sysnet to "Start qube automatically at boot". I sometimes tail /var/log/qubes/sys-idps. a. For me, it looks like this:

    # Poll for and then set up system gateway:
    /rw/config/qubes-openbsd-sysnet-script &

12. Set the "fedora-32-dvm" "NetVM" to "sys-net". a. This works around Qubes' 10.138.0.0/24 networking on disposable VMs. b. Repeat for all your DisposableVMs. c. Note: In Step 3 the "sys-net" Networking was set to "sys-firewall".

13. Thoughts and concerns: a. You have to manually manage a few of things: -- i. Match the LAN static IP address for "openbsd-21-sysnet" to what Qubes auto-assigns; conceivably, your actions could inadvertently change it. -- ii. Setting your static IP values used in the script. b. This requires editing the "openbsd-21" TemplateVM to make changes that persist in "openbsd-21-sysnet". -- i. My OpenBSD instance in the "openbsd-21" didn't save my root account changes until the third try. Just triple check that all your changes were saved if your "openbsd-21-sysnet" AppVM has issues. c. Several steps given above are only cautionary, or good practice. d. A couple of things that Qubes hard codes: -- i. TemplateVM updates to proxy via "sys-net". So I had to enable some services on "sys-net". -- ii. Disposable templates route through "sys-net". So, "sys-net" here forwards to "sys-firewall". e. During initial testing occasionally the hypervisor backend wouldn't release the PCI ethernet controller. Hence, Step 7.b. I haven't has any trouble since. f. This is a working integration guide and shell script. Edits are welcomed.

[1] https://github.com/unman/notes/blob/master/openBSD_as_netvm [2] https://groups.google.com/g/qubes-users/c/MpXLhz5COvM/m/smHA6a0VBgAJ [3] https://www.qubes-os.org/doc/standalones-and-hvms/ [4] https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-create.html

[tetrahedras]

On Wed, Sep 22, 2021 at 01:25:31AM -0700, Jay Holsapple wrote:

I have an OpenBSD net-vm up and running thanks to notes[1] and comments[2] from @unman. In outline format, my steps can be simplified (I think) to the following:

Thank you for the very detailed writeup!

[jcholsap]

@tetrahedras You're welcome. :+1: