Security implications of using two-way --pass-io vs. direct one-way input shell

As the title implies, it appears that there is a difference between using --pass-io and direct one-way shell input when communicating between domU's, or to dom0. If I got this wrong, please feel free to correct me.

As such, this issue is for discussion and awareness of this very topic here at QCC. It may therefore be important to be careful with --pass-io for all of us if we make scripts that interact between VM's, or even with dom0. It then may become a Qubes specific scripting trade-off question between practical uses, and security.

Appearance in functionality

--pass-io allows for greater utility and carries commands out in the other domU, and then carries it back into the original domU (or dom0). There may be some extra security features here that I'm not aware of, for example it'll only work around the same time the command was issued in the original domain? or perhaps it's not having such security barriers?
Direct one-way shell input to domains seem more secure, but again like above, this too is based on postulation. It is uncertain if this truly is "one way" and not just an alteration of the same function provided by --pass-io. It does however appear to be one-way signal, and therefore much more secure as it becomes akin to an analogue signal that cannot be easily hacked or exploited, and even if exploited, analogue signals in theory leaves very limited exploitable attack surfaces.

Please feel free to discuss, it'd be great if we can get to the bottom of this (feel free to correct any mistakes I did here too), and also for future references so that we know which is best suited for any future upcoming cross-domain communication job when making cross-domain Qubes scripts.

Reference examples

qvm-run personal xterm - runs xterm on personal

qvm-run personal xterm --pass-io - runs xterm and passes all sdtin/stdout/stderr to the terminal

qvm-run personal "sudo dnf update" --pass-io --nogui - pass a dnf update command directly to the VM

Source: @Jeeppler https://github.com/Jeeppler/qubes-cheatsheet/blob/master/qubes-cheatsheet.md

Did you do any more digging? Curious of the answer as well. qubes-devel might be the fastest approach, but looking through the source code might also work.

I did indeed dig further, and I believe I might have figured it out, but I still need to verify if its true or not, so it may still be relevant to confirm everything below.

Below is how I understand it Every time we run qvm-run or qrexec-client (of which qvm-run is heavily based), then it can be seen akin to a channel (not an official term). There can be multiple channels between the same VM's, and each channel can be considered secure isolations (I think can be considered secure isolation, this is a postulation).

These "secure channels" then can be considered the framework, on which rules are based to allow secure communication. The creation of "new" channels can be allowed/blocked by Qubes RPC rules, and by the looks of it, including or exluding --pass-io decides whether communication back and forth can be established (within the same channel).

So it seems to look like this;

Qubes RPC rules enforce whether new channels can be established between domains.
- Every new qvm-run (or qrexec-client) can be considered a new channel.
- RPC rules have nothing to do with what runs inside the channels, they only care about whether to allow new channels or not.
Security inside the channels then seem to depend on whether including or excluding --pass-io, which seem to decide if the target domain can talk back to the origin VM in spite to whatever the Qubes RPC rules are saying, since it uses the same channel already established.
- In other words, this can bypass the Qubes RPC rules, since the channel is already established, it only needs to communicate back to the target domain, within the same channel.

So if for example opening bash with --pass-io, it can send bash communication back and forth between the domains by using the established channel, but it can't create a new channel unless the RPC rules allow it.

In a sense, it looks like each channel is an isolation of its own, but in two different ways.

It can isolate the originVM by making sure the targetVM cannot talk back in the same opened channel.
It can isolate apps/bash/code etc. from other sources, i.e. domains cannot cross-over from one channel to another channel (presumably). 2.1. For example allowing one app in the targetVM to talk back to the originVM, would mean the attacker need to exploit the app in the targetVM in order to use the channel on which the targetVM's app is using (and allowed) to communicate with the originVM. 2.1.1. This is probably where using --pass-io may (presumably) become dangerous. Channels are probably isolated, but could an attacker exploit the app/code which is allowed to communicate with the originVM, and thereby invade and bypass the isolation?

Running sudo bash -x /usr/bin/qubes-dom0-update also is a good example, it prints in the channel what happens, and you can see the --pass-io command in action in a semi-transparent way. It appears to be what carries over the output from the dnf update command in the target VM, transferred via --pass-io to dom0.

Speculation - Could an attacker maybe compromise dom0 by compromising dnf bug in the target updateVM?

Some extra questions emerges. For example, can these "channels" be considered truly secure? For example, just like how malware and hackers can insert code into apps, websites or online traveling signals at every compromised router, so too, could an existing on-going Qubes qvm-run channel be compromised in similar ways? Are they truly isolated?

What if running a script between two domains with qvm-run-in-vm someVM --pass-io bash, then between two domains, could a Malicious attacker insert anything into one of these domains to attack the other domain? What if the target less trusted domain gets compromised, can the attacker then use the open channel to bypass the Qubes RPC channel security rules the same way they bypass regular firewalls in networking? After all, RPC rules seem very much like the same logic as firewalls.

Speculation - Maybe restarting a VM once in a while is a good idea for this reason too? Highly speculative without any solid basis though.

Another question emerges, even if for legit uses, can channels communicate with each others at the origin domain, or the target domain? It seems like it can't, and it's probably good too since this seems like a whole new can of worms that allow exploiting to bypass the isolation. Because of that it seems like channels are strictly isolated both at target and origin domain, but it still remains an unanswered question for now.

Definitely have some uncertainties, but I think the above is getting closer to how it works, but it remains to be verified though. It is definitely a good idea to look in the code itself or to ask on qubes-devel though, the above is kind of like what one does when reverse engineering technology without knowing the code/tech, and that can potentially be full of flaws right :/ But in so far, maybe we can ask better questions the more we understand before asking though.

Maybe we can establish some practical examples, for example to figure out how a program (or bash script) can communicate back and forth on each VM. Presumably it would require the program to be designed to read and use the output of another program, whether the other program was made for this purpose or not.

For example, running sudo bash -x /usr/bin/qubes-dom0-update in dom0, seems like a really good example to reverse engineer. For example;

Preparing folders etc. in the updateVM

echo 'Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...' Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...

qvm-run --nogui -q -u root sys-firewall 'mkdir -m 775 -p /var/lib/qubes/dom0-updates/'

qvm-run --nogui -q -u root sys-firewall 'chown user:user /var/lib/qubes/dom0-updates/'

qvm-run --nogui -q sys-firewall 'rm -rf /var/lib/qubes/dom0-updates/etc'

tar c /var/lib/rpm /etc/yum.repos.d /etc/yum.conf

Above are examples of 3 channels that can only go one way, dom0 sends simple instructions to domU. Then;

qvm-run --nogui --pass-io sys-firewall 'LC_MESSAGES=C tar x -C /var/lib/qubes/dom0-updates 2>&1 | grep -v -E "s in the future"'

I'm not fully sure, but this line appears to be what selects what needs to be copied from all the packages being downloaded via the updateVM, progressively, as they happen. Strictly speaking, I'm a little unsure about this one, it might only be part of the logic, with the other part somewhere else in the qubes-dom0-update binary script.

and

qvm-run --nogui --pass-io sys-firewall 'script --quiet --return --command '\''/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui --exclude=qubes-template-debian-9,qubes-template-fedora-28, '\'' /dev/null'

Appears to be taking the text output in the updateVM and carries it over to the dom0 terminal by using --pass-io to allow back and forth communication, and the script command to print a copy of output in dom0 terminal.

It would be flawed logic to say that just because --pass-io is used, that it then is needed to allow back and forth communication. But it does indeed look like it, and the available commands and options being used in the update proxy appear very limited, so it doesn't seem farfetched to rule out other possible explanations. But I don't have enough insight to do that thoroughly and fully though, I can't rule out all other possibilities. But enough can be ruled out to at least make it look like there are no other possibilities, hence it appears like this is how --pass-io work in practice, and it also matches the manual description, here

qvm-run --pass-io (Pass standard input and output to and from the remote program.)

Source: https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-run.html

Something else worth considering is also that it appears not only safer, but also gives more flexibility to use --pass-io in addition to RPC rules, rather than strictly only relying on RPC permissions. I think this might (maybe) be, in part, one of the reasons the developers came about the naming of "less and more secure domains". If both have the same RPC permissions, then no VM is less or more trusted over the other VM. Consider this 5-steps;

If both VM's can talk to each others with the same RPC rule permission(s) when communication is needed back and forth, then this is more insecure and has a larger attack surface than compared to, say using, qvm-run pass-io in such a way, that the target VM is blocked in its RPC permission rules, however the originVM is not blocked in the RPC permission rules. Only then, and only when, the originVM initiates the channel (and including the --pass-io option), is the targetVM then allowed to communicate with the originVM.
This then provides extra isolation in such a way under the assumption that all RPC rules are correctly blocked, that the originVM can mostly only compromise itself if it uses the --pass-io, because almost every other RPC is blocked without an open channel from the originVM.
This then means RPC rules can be more strict by default without loosing out on flexibility, and in addition to that also provides the basis for "less and more" trusted VM's, and its isolation principles.
The --pass-io then, along with the RPC rules, appears to be very central pieces to the mechanics of inter-VM Qubes isolation.
In addition, excluding --pass-io also makes it possible to make really strict isolation between the more secure, and less secure domains. For example if the originVM does not initiate any qvm-run --pass-io commands, and the RPC is fully locked down, then it's pretty much isolated fully, while still being able to control the less secure targetVM (unless a bug in Qubes itself is exploited).

Excluding the use of --pass-io when it isn't needed, then appears to strengthen and enhance the Qubes RPC isolation system, but even if including the --pass-io option it still holds isolation value, because the moment the channel is shutdown, so too is the RPC permission. But changing RPC permissions is more permanent (until you change them again yourself, that is), and hence using pure RPC permissions is not only less isolation if compared to having to do the same tasks as when using --pass-io, it's also less flexible in practice to only use RPC permissions. --pass-io then provides flexibility to RPC rules, and also allows RPC rules to be more strict without having to worry about the flexibility. In other words, --pass-io then appears as a method to strengthen RPC rules further, and when using --pass-io, it can momentarily allow to bypass RPC rule permissions for a specific task (rather than opening it for everything else as well), and only as long the channel is kept open.

And finally, it appears that the --pass-io option allows the RPC rules to be more strict out-of-the-box, since --pass-io can just be included in the more secure domain, whenever needed, and --pass-io only works if the originVM is allowed to establish a channel anyway, so the --pass-io isn't universal unless it can establish a channel first. So there also appears to be a hierarchy, RPC rules > --pass-io, although that might be kind of obvious at this point.

Conclusion Calling --pass-io dangerous might only be one side of the coin, since it's also what gives Qubes extra isolation at the same time (when inter-VM communication is needed). It may therefore be more accurate to say that --pass-io should be used responsibly, rather than out-right discouraging the use it.

Please let me know if you find any flaws in any of this :) It's an on-going learning process, mistakes might have been made.

My understanding of RPC rules is the same as yours, so I think your last comment makes sense! You've done more research on it than me though, so I will trust your interpretation.

Don't be so quick to say that though, I'm sure I need to refine my understanding, and you may have a better understanding than me, so feel very free to correct it if you spot anything :)

I will definitely read more up on it in the future though, it's one of those aspects about Qubes OS that seem important to understand, and I also suspect you might feel the same way about it.

Note: Closing this issue for now though, due to issue inactivity, it can be opened again in the future if needed.

Qubes-Community / Contents

Security implications of using two-way --pass-io vs. direct one-way input shell #48