Idea: dynamic firewall

[Rudd-O]

The Qubes firewall is an amazing piece of technology integration. That said, it happens to be the case that many services these days change how traffic is routed to them using DNS, rendering it almost impossible to block based on DNS.

For example: I boot up a VM that will run an app, which connects to mxs-services.whatsapp.net on port 443. If I configure this on the Qubes firewall, it will work... for the next thirty minutes or so. Then the DNS response changes -- the app requeries the DNS records, it receives a completely different set of DNS responses (most likely in a very different subnet), the Qubes firewall begins replying to TCP SYNs with ICMP admin prohibited, and the app simply stops working correctly. Endgame.

The alternatives are to continually add new netblocks to the firewall configuration as they are discovered by manual inspection using a tcpdump terminal, or to completely abrogate security and open the machine up to leaks by allowing any and all traffic to the destination ports (most services these days just use port 443).

Here's what I would like to see:

• A service (associated to a firewall VM setting, and defaulting to off) that snoops on outbound DNS requests and incoming DNS replies from AppVMs attached to the firewall VM it's running on.
• This service maintains an up-to-date cache of (host name -> IP addresses) sets.
• This service also communicates with the Qubes firewall running on the machine.
• Whenever the cache for a host name known to the Qubes firewall changes, it asks for the Qubes firewall to reset its rules
  • Alternatively, more correct and less racy, but harder: the service alters the firewall configuration by (1) crawling it, (2) deduces the host names from the IPs listed on it and expired versions of its cache, then (3) generates a new firewall configuration with the new non-expired IPs substituting the old expired IPs (4) applies the firewall configuration.
• The service runs properly hardened, dropping any and all privileges it does not need.
• The service is written in a memory-safe language like Rust.

This would immensely improve the user experience of the Qubes firewall compared to the current system. It also has potential uses in split-DNS scenarios where a corporate user may or may not be logged into a corporate network some of the time, so the firewall really does need to respond to changes in network configuration.

Ideas for extension

Once the basics are in place, it should be possible to make the DNS firewall behave interactively. Imagine an incremental improvement whereby an AppVM not currently authorized to connect to host.name.org attempts to first resolve the address, then receives the response (duly-cached by the service I'm proposing). Now, the service sends its first TCP SYN. In principle, the service can detect this and present an "Outbound connection attempt prompt" (how best to wire this interactive protocol into the Qubes qrexec framework is left unspecified here), and then the user of the machine can say "Yes / No / Yes, always", upon which the firewall makes the policy decision of ignoring further TCP SYNs or allowing the connection to be established by adding the necessary rules and letting the AppVM retry the TCP SYN. The AppVM would not know this has happened -- the only side effect is that the connection will take longer to be finished due to exponential backoff.

Pitfalls

I have not yet considered if the service should independently snoop each VIF to separate how it'd change firewall rules (seems more secure that way). It might end up being necessary anyway, since it's important for such a service to maintain a per-VM high watermark of in-flight (not yet deemed complete) DNS requests, else the service be DDoSable by a misbehaving VM.

Project management

This seems worthy of at least a bit of funding. I believe I can pony up some. We can negotiate based on the effort estimated to get here.

(If any of this could be implemented as a Mirage unikernel instead, or possibly a Unikraft unikernel -- which means it's not gonna be limited to OCAML -- so that it became a substitute for the standard Linux firewall VM offering of Qubes, I'd be even more interested in this project.)

[marmarek]

Actually, a feature like this is already in progress for qubes-mirage-firewall. cc @yomimono @linse

[andrewdavidwong]

Is this a duplicate of #3641?

[Rudd-O]

Oh! Awesome! Where can I read the code in progress??

On August 8, 2019 3:50:35 AM GMT+02:00, "Marek Marczykowski-Górecki" notifications@github.com wrote:

Actually, a feature like this is already in progress for qubes-mirage-firewall. cc @yomimono @linse

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/QubesOS/qubes-issues/issues/5225#issuecomment-519330969

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Rudd-O]

The bugtracker only mentions supporting dynamic Qubes rules (what we have today), nothing else.

On August 8, 2019 3:50:35 AM GMT+02:00, "Marek Marczykowski-Górecki" notifications@github.com wrote:

Actually, a feature like this is already in progress for qubes-mirage-firewall. cc @yomimono @linse

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/QubesOS/qubes-issues/issues/5225#issuecomment-519330969

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[yomimono]

Since I've intended to write you a long and thorough comment for some time but not succeeded in doing so, here is a short and scattershot one: the work @linse and I are doing is based off the existing qubes-mirage-firewall (previously just a NAT device), which you can find at https://github.com/mirage/qubes-mirage-firewall and has a writeup at https://roscidus.com/blog/blog/2016/01/01/a-unikernel-firewall-for-qubesos/ .

Our version dynamically reads the Qubes 4 rule language from QubesDB and updates the ruleset accordingly (which applies to outgoing client traffic not corresponding to any existing entries in the NAT table); on changes to the ruleset it will drop the NAT table as a measure to ensure that previously-allowed connections are not incorrectly allowed to continue. It does not currently drop any entries in response to changes in name resolution results.

Our strategy for DNS has been for the firewall to resolve names itself when necessary and maintain its own cache, since snooping client DNS traffic may become untenable with strategies like DNS over HTTPS. Snooping DNS could be added as an optimization when available. It's probably necessary to handle round-robin correctly.

If you're interested in discussing in more detail, let us know what specific bits you're interested in; we're happy to talk about the design further.

[DemiMarie]

@yomimono To handle DNS round-robin correctly, it is necessary to do the following:

• Refuse to resolve (with SERVFAIL) domain names that a VM is not allowed to access.
• Do not allow access to IP addresses that the firewall has not itself resolved.
• Optionally, return fake addresses to clients, so that the firewall can track connections to different domains that resolve to the same IP address.

Honestly, a filtering SOCKS5 proxy may be a better solution in the long run. It has none of the problems that an L4 firewall does.

[rugk]

I came here, because I saw the Prototype Fund project for qubes-mirage-firewall (German). If I read it correctly, the disadvantage of the old Firewall is that it uses too much RAM (350 MB). The contribution should go back into the "QubesOS-Contributions-Repository". (whatever that is)

The code and blog post go into more detail (I've only skimmed), and the repo explains this is:

A unikernel that can run as a QubesOS ProxyVM, replacing sys-firewall.

So what is the future of this project? Will it be included by default in Qubes OS? Can we get it to replace sys-firewall? Was that even the aim? Or has this idea been dropped? (If so, why?)

I just would like to know the status of this project and this was the only issue, where this project has been mentioned by @marmarek. After all, if properly included, that may be a good contribution to Qubes OS. Just now it looks a little academical as a "proff of concept" or so, as I doubt any Qubes user will clone a random proxy VM into dom0 and just execute it. Especially if the last commit is from 2017.

/cc @yomimono

[xaki23]

After all, if properly included, that may be a good contribution to Qubes OS. Just now it looks a little academical as a "proff of concept" or so, as I doubt any Qubes user will clone a random proxy VM into dom0 and just execute it. Especially if the last commit is from 2017.

a) qubes-mirage-firewall is actualy pretty much production grade for the basic "outbount nat fw" usecase for a longish time, acting as download proxy for dom0 is not supported at all, and any kind of custom fw rules is awkward. b) there is no need to "clone a random proxy into dom0": https://github.com/QubesOS/qubes-builder/blob/master/example-configs/kernel.conf c) if you look at the repo you linked, you can see where it was cloned/forked from, and if you look at that you will notice there have been a pile of commits in the last weeks: https://github.com/mirage/qubes-mirage-firewall/commits/master d) including an open PR for dynamic rules (where i am not sure how much of that matches the ideas here): https://github.com/mirage/qubes-mirage-firewall/pull/96 e) but at the moment i do not see qubes-mirage-firewall as a suitable main-fw for non-technical qubes users, but if you are not afraid of using a commandline i can highly recommend trying it!

[andrewdavidwong]

@rugk:

The contribution should go back into the "QubesOS-Contributions-Repository". (whatever that is)

You can read more about this in our documentation on Package Contributions.

[rugk]

Ahh thanks for the replies.

Did not see it has been merged in here, that explains a lot and there is also progress. Also thanks for the PR link, that looks good.

So the only thing missing is the inclusion into the Qubes UI, so that "non-technies" can use it? And propbably then sys-firewall can be dropped? (as it is replaced by that)

[brendanhoar]

So the only thing missing is the inclusion into the Qubes UI, so that "non-technies" can use it? And propbably then sys-firewall can be dropped? (as it is replaced by that)

...dropped and then replaced by sys-dom0-update-proxy or something similar to handle the dom0 updates role :P

Brendan

[Rudd-O]

What's the status on productionizing the Mirage firewall to read and act upon the Qubes DB firewall rules set up by admin-core?

[linse]

Hi, you can try out the qubes-mirage firewall for qubes 4 and the latest mirage libraries by checking out the squash branch https://github.com/roburio/qubes-mirage-firewall/tree/squash and running mirage configure -t xen && make depend && make. There is also a pull request which is still under review. We're happy to hear how it works for you.

[0spinboson]

So the only thing missing is the inclusion into the Qubes UI, so that "non-technies" can use it? And propbably then sys-firewall can be dropped? (as it is replaced by that)

...dropped and then replaced by sys-dom0-update-proxy or something similar to handle the dom0 updates role :P

Brendan

if that VM is offline at all other times, that's still helpful to people who are ram constrained. :)

[linse]

We just updated the PR https://github.com/mirage/qubes-mirage-firewall/pull/96#issuecomment-615226289

[Rudd-O]

Now all the PR linked above needs, is a feature that updates the host-based firewall rules based on DNS TTL, and boom, it works.

Not the same as this feature request, but at least it's closer.

[DemiMarie]

@Rudd-O That works so long as the firewall proxies DNS requests.

[Rudd-O]

No. It only needs the firewall to watch DNS requests and replies, as they go past the networking bus. There need be no active network proxying involved. It can be accomplished by a simple pcap daemon.

On May 18, 2020 7:02:25 PM GMT+02:00, Demi Marie Obenour notifications@github.com wrote:

@Rudd-O That works so long as the firewall proxies DNS requests.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/QubesOS/qubes-issues/issues/5225#issuecomment-630314487

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[marmarek]

That's the theory. The practice unfortunately is not that bright unfortunately. Applications can ask different DNS servers if firewall allows (and you can't simply trust that answer, as it would allow trivial firewall bypass). It's getting even harder with things like DoH or DoT. So, in practice, just watching traffic may not be enough (or rather - will be enough in only some cases, but will totally break in others).

[DemiMarie]

The usual practical answer is a SOCKS or HTTP proxy, but those only work for TCP. I have a configuration based on one posted years ago to the mailing list, which works well for me.

[Rudd-O]

The Qubes DNS model totally breaks with DoH anyway, so we can rule that out.

As proposed earlier, a daemon that watches and caches DNS requests and replies, then asks the user whether to allow or deny access to those addresses upon first connection attempt, is what the feature request is about. Think of it as a setting in a Qubes firewall rule that is "ask", in addition to allow and deny.

On May 24, 2020 4:19:13 AM GMT+02:00, "Marek Marczykowski-Górecki" notifications@github.com wrote:

That's the theory. The practice unfortunately is not that bright unfortunately. Applications can ask different DNS servers if firewall allows (and you can't simply trust that answer, as it would allow trivial firewall bypass). It's getting even harder with things like DoH or DoT. So, in practice, just watching traffic may not be enough (or rather - will be enough in only some cases, but will totally break in others).

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/QubesOS/qubes-issues/issues/5225#issuecomment-633166955

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[DemiMarie]

For this to work, it is necessary for DNS requests to non-whitelisted domains to be blocked.

[Rudd-O]

On May 25, 2020 8:40:00 AM GMT+02:00, Demi Marie Obenour notifications@github.com wrote:

For this to work, it is necessary for DNS requests to non-whitelisted domains to be blocked.

Not really. The ticket isn't about DNS filtering. It is merely about letting the user choose whether to add allow rules for domains that correspond to IPs that their AppVM wants to connect to. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[DemiMarie]

On 2020-05-26 05:09, Rudd-O wrote:

On May 25, 2020 8:40:00 AM GMT+02:00, Demi Marie Obenour notifications@github.com wrote:

For this to work, it is necessary for DNS requests to non-whitelisted domains to be blocked.

Not really. The ticket isn't about DNS filtering. It is merely about letting the user choose whether to add allow rules for domains that correspond to IPs that their AppVM wants to connect to.

Without DNS filtering, it is quite likely that the firewall will be subverted by accident, especially when many sites of varying trust levels share the same IP addresses. In the era of CDNs and Cloudflare, that is quite common, which is why I use a filtering HTTP proxy instead.

Sincerely,

Demi

[DemiMarie]

Even with DNS filtering, there is still the issue of DNS over HTTPS/TLS/etc. I strongly recommend switching to a hybrid SOCKS5/HTTP proxy, where “hybrid” means that both are supported, preferably on the same port.

[marmarek]

@DemiMarie this is already tracked in https://github.com/QubesOS/qubes-issues/issues/5032, responded there.

[3hhh]

Why not just pin DNS responses for downstream VMs to the DNS response that was originally obtained during firewall setup?

I.e. essentially cache the response for the lifetime of sys-fw or the TTL of the DNS response? Especially since Qubes OS does a lot of DNS DNAT anyway and there should be quite a few open source DNS caches around?

Passively listening to potentially attacker-controlled DNS requests sounds more dangerous than controlling the DNS requests oneself.

[3hhh]

For example, one could deploy dnsmasq inside sys-fw or even some dedicated VM, then Qubes OS could write the domain/IP combinations resolved during firewwall setup to /etc/hosts (used by dnsmasq) or just pass them to dnsmasq via --adress during startup.

If necessary, one could even write a little service that updates /etc/hosts after a while.

P.S.: Just noticed that instead of /etc/hosts, one would probably have to go with the hostsdir option or send SIGHUP on updates.

[DemiMarie]

dnsmasq has too much attack surface for us to use, sadly. Unbound would be a better choice, but something written in a safe language would be strongly preferred.

[3hhh]

dnsmasq has too much attack surface for us to use, sadly. Unbound would be a better choice, but something written in a safe language would be strongly preferred.

Hm so this issue boils down to finding the right software.

Honestly I don't see much of a difference in risk wrt dnsmasq or unbound when looking at their CVEs: unbound vs dnsmasq

Neither of them looks very good. Considering the very limited use case (caching for a fixed set of domains) almost none of the CVEs would have affected Qubes users (e.g. all cache posoning attacks would have failed at the firewall layer and cause DoS only; only RCE from DNS requests or responses would hurt).

Anyway I wouldn't recommend using some niche software either (e.g. go-dnsmasq just because one considers the language more safe or so) as simply no one ever looks at their security and usually no one maintains them either. If there's a bug in often-used software such as dnsmasq or unbound, there'll usually be a fix around in no time.

(Disclaimer: I run several dnsmasq and unbound instances myself.)

[3hhh]

I implemented the DNS pinning idea and it appears to be working.

Nowadays even systemd ships its own DNS server, so I used that (systemd-resolved) for the purpose. The aforementioned ones should work in similar ways though.

Whether it's worth the increased attack surface I'm not sure. At least the impact is limited with disposable firewall VMs.

It might be worth adding it as an optional feature though.

[rwiesbach]

As there are no recent updates to this ticket and quite a bunch of posts: What is the current state of things regarding DNS-based blocking in qubes firewalls and changing IPs for configured hostnames? I myself have issus with my email vm, because my provider changes the IPs of IMAP/SMTP (probably for load-balancing) and because add-on updates of thunderbird require a connection to a mozilla domain which has different IPs as well.

Thank you.