Shared folders (perhaps over qrexec?)

[tonsimple]

Basically what it says on the can.

Right now, I'm using a separate VM with smb server to share folders between several VMs (including a windows one), but it 1) has unpleasant overhead 2) smb has a lot of features that are, well, not really needed for this usecase and are just extra attack surface/general drag 3) the VMs have to share a firewallvm/proxyvm

inspired by reading about https://github.com/QubesOS/qubes-app-linux-usb-proxy while trying to come up with a way to share a folder with a qube that is supposed to have direct connection to netvm (thus not connected to the proxyvm which the smb hosting one is connected to)

[cyrinux]

Hi, What about this?

https://groups.google.com/forum/m/#!topic/qubes-devel/rCc20BTZ9co

[tonsimple]

@cyrinux Thanks! Looks interesting. Though comments by Vít Šesták (particularly about inode number guessing) worry me a mite.

Also, I don't quite understand what "The home directory of the destvm should be readonly accessible in the given directory in the srcvm" means, I guess I'll ask in that project's github issues.

[jpouellet]

Can you help me understand the intended use case?

"Shared folders" in every way I've seen implemented are dangerous to some degree (in any case more than qvm-copy/move-to-vm), so I'm curious if perhaps the root motivation can be satisfied in another manner. I really don't see what purpose it serves, but you're not the first one to ask for this or hack together a solution, so there must be something I'm not seeing...

[adrelanos]

Does https://www.qubes-os.org/doc/copying-files/ work for you?

---

Otherwise, let me throw in a big hack. Perhaps ssh over qrexec and then using sshfs.

[tonsimple]

@jpouellet Say you're downloading a large (like, 4 TB large) file (torrent or whatever) and you can't commit 4TB both to VM that is "internet-facing" and the VM that is supposed to store/process/etc the file.

Right now I'm doing a samba share, but samba is hugely unpleasant in ways I outlined in original post.

I realize that shared folders add attack surface and complexity no matter how you slice them. But I would prefer things that add less of that than "a dedicated SMB server"

@adrelanos Technically copying files would work if I could commit enough storage to both the VM that "does the downloading" and the VM that is "supposed to eventually handle/process/etc. the file"

SSHFS is definitely a viable alternative at least in the sense that it's probably less complex than a "fully grown" SMB, but point 3 and to some extent point 1 from my original post would still stand.

Ideally, the shared folder solution would not need to have the "host VM" (source of share) and "destination vm" (the machine to which the folder is shared) to have same proxyvm/firewallVM (so that the machine doing the "processing" of the file could be left with no "proper networking" beyond the single folder share)

[jpouellet]

Ideally, the shared folder solution would not need to have the "host VM" (source of share) and "destination vm" (the machine to which the folder is shared) to have same proxyvm/firewallVM (so that the machine doing the "processing" of the file could be left with no "proper networking" beyond the single folder share)

This is achievable with ssh over qrexec. I use this on my Qubes server. Create a qrexec service in the target VM which starts sshd in inetd mode (sshd -i), and then use ssh '-oProxyCommand=qrexec-client-vm other-vm-name local.SSH'. It should be noted that from a security perspective this is equivalent to qubes.VMShell and effectively 100% eliminates any meaningful trust boundary protecting the target VM from the source VM. It is perhaps even more dangerous than qubes.VMShell since by default ssh exposes a source domain pty to manipulation by the target domain, which qubes.VMShell avoids. If this is really* what you need, then maybe give it a try.

However: This completely violates the Qubes security model. If you are in the situation where you really need to be sharing such giant data across VMs, then perhaps using a single VM with the tools of both installed makes much more sense. Unless someone can provide evidence that users really require such functionality and are currently doing something more dangerous than this, then I believe it would be a design mistake to make this functionality easily available.

[tonsimple]

@jpouellet It seems to me that this approach would possibly be more of a "risk" than having source VM "doing samba" to share a folder with "downloader" (with some basic firewall rules preventing direct internet access for the source VM)

So far, SSHFS seems by far the nicest of options.

[jpouellet]

Right, but then why have multiple VMs in the first place? If doing so eliminates the trust boundary between the VMs, then you gain no security by having separate VMs, so why use separate VMs at all? Do you need to use multiple tools on the same large dataset which are only available on linux XOR windows or something?

I don't see the benefit to further decomposition in this case -- and I say that as someone with an absurd number of VMs myself.

[tonsimple]

@jpouellet Well, I suppose samba/SSHFS maintains some security boundary (adjusted for issue of exploits in SMB (lol ;-) ) or (in case of sshfs) SSH.

An "ideal-hypothetical" shared folder solution would be some daemon that doesn't need to run as root and can be run as separate user (which should in theory limit the amount of mayhem if "something happens") but I'm not sure that's possible given current qrexec mechanisms.

[marmarek]

IIUC the problem is:

1. You have huge file downloaded from the network
2. You don't want the tools used to parse the file to communicate with the network

I think with the currently available features, solutions closes to those requirements is:

1. Create fresh AppVM (ideally DispVM, but it isn't possible to adjust its size, at least right now)
2. Start it with network connected
3. Download the file
4. Disable networking for this VM (set netvm to "none")
5. Process the file.
6. Do not ever connect this AppVM to the network again, if you want to process another file, use new AppVM.

[andrewdavidwong]

Sounds like a similar use case to https://github.com/QubesOS/qubes-issues/issues/2486#issuecomment-282463102.

[holiman]

I also would like this feature, but for a different usecase. I use different VMs for development, e.g one for golang, another VM that builds docker images, and a third for npm/electron and javascript things. I would like to be able to have a shared disk area that I can use for git repositories, instead of having to qvm-copy entire repos between the different VMs, and also not have to set up gpg-split in each vm (I use gpg-split to sign git commits).

I'd want to do e.g electron-dev in a, build docker image in b and commit/sign/push in c.

[jasperweiss]

If VM's could have a shared directory, it would open up a lot of use cases. For example, one might use a tool such as Cryptomator to encrypt files on the shared directory while using a separate networked VM to run the cloud sync client. That way the plaintext files and encryption keys are never exposed to a networked VM running potentially untrusted sync clients.

The virtual drive exposed by cryptomator could even be mounted on a different VM so that a potentially malicious file can't leak the files or keys by putting them in the shared directory. See visualization here

[acertain]

With complex policy stuff, this should be able to be more secure. e.g.:

• when a vm tries to read/write a path, prompt the user, with options for allowing all files in a subfolder
• when allowing a write, have a checkbox for invalidating other vm's read permissions
• log & display which vms touched the path in the prompt

[adrelanos]

When showing the path in dom0, is the path (a string in programming) itself a security risk? A path including special characters or being super long could be exploited to do weird things. Just something to keep in mind when programming this.

[AshleyYakeley]

A couple more of scenarios where this would be useful:

1. I have an ansible script to set up VMs. Obviously it must run in dom0. I put it in a directory shared with an AppVM, and in the AppVM I use Atom or VSCode to edit it and push changes to a remote git repository. I can test it as I make changes, without having to laboriously copy it to dom0 each time.

2. I have a KeePass file of secrets that I need synchronised on multiple devices, so I share it over a cloud service (e.g. NextCloud). On Qubes, I have an AppVM with network access that synchronises the remote file to a local file. This local file is in a directory that is shared with another "vault" AppVM, that has no net access. The file is only ever decrypted in the "vault" VM.

[marmarek]

1. I have an ansible script to set up VMs. Obviously it must run in dom0. I put it in a directory shared with an AppVM, and in the AppVM I use Atom or VSCode to edit it and push changes to a remote git repository. I can test it as I make changes, without having to laboriously copy it to dom0 each time.

@woju has written git over qrexec (with built-in signature checks and integrity validation) specifically for this purpose: https://github.com/woju/qubes-app-split-git

2\. I have a KeePass file of secrets that I need synchronised on multiple devices, so I share it over a cloud service (e.g. NextCloud). On Qubes, I have an AppVM with network access that synchronises the remote file to a local file. This local file is in a directory that is shared with another "vault" AppVM, that has no net access. The file is only ever decrypted in the "vault" VM.

This could be useful case indeed, but honestly I think exposing full filesystem semantics would be risky here. Compromised VM could for example try to modify the file while KeePass is accessing it, in attempt to exploit some TOCTOU type KeePass bug. I'm not saying there is one - just saying there is a lot more such AppVM could try with fully shared folders.

A safe(r) mechanism should ensure that only one VM has access to any file/directory at a given time. Something we're very careful in qubes.Filecopy service. Technically it is possible to build something around qubes.Filecopy service - for example have a wrapper around KeePass that copies a database out of ~/QubesIncoming at the start and after closing sends it via qvm-copy-to-vm to a network-connected VM. The way I describe it does require writing some script, but when done properly, should be significantly more secure than "shared folders".

[AshleyYakeley]

Hmm, rather than shared directory, what about some kind of automatic synchronisation similar to rsync?

[marmarek]

That would be a better idea.

[woju]

On Wed, Oct 23, 2019 at 06:15:00PM -0700, Marek Marczykowski-Górecki wrote:

1. I have an ansible script to set up VMs. Obviously it must run in dom0. I put it in a directory shared with an AppVM, and in the AppVM I use Atom or VSCode to edit it and push changes to a remote git repository. I can test it as I make changes, without having to laboriously copy it to dom0 each time.

@woju has written git over qrexec (with built-in signature checks and integrity validation) specifically for this purpose: https://github.com/woju/qubes-app-split-git

I wouldn't object if you appropriated it under QubesOS/, packaged and released it.

[holiman]

Having cross-vm git integration would be awesome, but another thing that would be really nice would be to have qvm-sync, similarly to how we already have qvm-copy. qvm-sync would function llke rsync, but could use QubesIncoming/<sourcevm>/sync/ as the destination root.

[holiman]

So, I went and implemented this idea. It lives under https://github.com/holiman/qvm-sync . It's still a bit of work in progress, but it seems to work pretty ok. When I sync my dev tree (go-ethereum) with my docker builder qubes after checking out a different branch, the sync completes in 2 seconds, instead of like 30.

Feedback is welcome

[andrewdavidwong]

So, I went and implemented this idea. It lives under https://github.com/holiman/qvm-sync . It's still a bit of work in progress, but it seems to work pretty ok. When I sync my dev tree (go-ethereum) with my docker builder qubes after checking out a different branch, the sync completes in 2 seconds, instead of like 30.

Feedback is welcome

@marmarek, should we turn this into a [Contribution] community dev issue?

[Coeos3]

I also use Keepass and Cryptomator with cloud storage. While for Keepass a qvm-sync function would work fine, it wouldn't work for big folders when disk space is an issue. Could there be something like a "storage VM" that we could attach to a qube whenever needed?

To use the Cryptomator example (containing tons of small files), I could then attach it to the internet facing VM to sync with the cloud, detach it, attach it to an offline qube where I decrypt and modify the files (potentially with a dispVM), and finally encrypt, detach, and re-attach to the internet facing VM to sync the changes.

1. Would that have a smaller attack surface than running Cryptomator on the internet facing VM?
2. Is there already a way to achieve something similar?

[tabbyrobin]

I think there's utility for some kind of "qvm-share" tool (for read-only access, and/or also for read-write access). Which does not copy the data. (But as yet, i don't have any ideas on how to securely implement read-only sharing without copying...)

qvm-sync looks really great! I think it will be really useful. But if I understand how it works, it uses the same qvm-copy mechanism under the hood. So it does not solve the limited space issue (still makes double copies of whatever is shared).

Related, I second @Coeos3 on suggesting some kind of "storage VM" functionality. I think in some cases centralized storage can be a useful paradigm (it's one that's used on Android with the sdcard permission access). Especially for saving space.

I think a "qvm-share" tool could allow users to set up their own centralized (or non-centralized) storage easily.

Example use case: On my computer i have a 200gb ssd and a 1 tb spinning disk. I keep a ton of data on the 1 tb spinning disk "offline" (only access it thru a dedicated app qube "a-storage".) Some of that data, for example music library or a Wikipedia archive (100gb zim file), i want to use in certain other qubes (music player, Kiwix), but don't want to copy off the 1tb drive. Sometimes this is for space usage, sometimes because i don't want to end up with two diverging copies.

[Coeos3]

@sptankard : FYI, lacking a better solution I ended up creating a virtual disk image which lives in a storage-VM and which I can attach to any other appVM. It's not great from a security perspective as @marmarek explained above. Apart from that it works fine, have been doing that the last 2 months.

[tabbyrobin]

@Coeos3, i'm curious how you attach the image to another VM? (What commands/tools/is this generic xen utilities)

Another workaround i have read about (haven't tried) is to create a secondary lvm storage pool, and store some appVMs on a different disk from the main one: https://www.qubes-os.org/doc/secondary-storage/

Obviously none of these solutions are ideal, which is why i do think there's utility for some kind of non-copy qvm-share feature. The issue seems to be how to do it securely. Per my understanding, making it read-only (for the receiving end), as opposed to read-write, takes care of a bunch of issues. But there's still a lot of issues around how much filesystem semantics to include in the qubes-rpc calls and how to appropriately sanitize them. Tho i imagine it's not an issue without precedent as there are a lot of networking remote fs protocols...

[Coeos3]

(I had found the original instructions on a forum, don't remember where though) The steps are:

• create a storage VM, in which you use dd to create a disk image, and then losetup to create a virtual device. Create partitions and filesystem.
• attach to target VM using Qubes Devices applet or dom0 console (qvm-block) and mount it there.

I then created some bash scripts in dom0 to do the attaching, mounting and vice-versa for the VMs I use with that storage (dom0 just sends the commands to the VMs, I'm not mounting anything there).

In more detail, these are the commands to use:

1. create a storage_VM Creating virtual disk image (in storage_VM Terminal): sudo dd if=/dev/zero of=/home/user/storage.img bs=1M seek=20480 count=0 the number after seek is disk size in MB.

2. creating virtual device from disk image (in storage_VM Terminal): sudo losetup -f --show /home/user/storage.img

3. Attaching disk image to a VM (in dom0 Terminal): qvm-block attach VMname storage_VM:loop0 (or whatever VMname and loop it is)

4. Identifying disk image (in VM Terminal): sudo fdisk -l

5. Creating partition (in VM Terminal): sudo fdisk /dev/xvdi (or whatever you identified above)

6. Creating filesystem in partition (in VM Terminal): sudo mkfs.fat /dev/xvdi1

7. Mounting volume (in VM Terminal): the first time in that VM only: sudo mkdir /mnt/removable (or another name. Use sudo chown -R user /mnt/whatever_name to give write access) sudo mount /dev/xvdi1 /mnt/removable

8. Unmounting volume (in VM Terminal): sudo umount /dev/xvdi1

9. Detaching disk image from VM (in dom0 Terminal): qvm-block detach VMname storage_VM:loop0 (or whatever VMname and loop it is)

The steps in bold is what you need to do each time, the others only the first time.

[iamahuman]

Workaround: https://github.com/unman/qubes-sync

[iblue]

I wrote a tool to mount directories across Qubes. It can be found at https://github.com/iblue/qvm-mount. It works, the performance is better than expected (~500 MB/s on my machine) but it's still in an alpha stage, so not tested that well.

[DemiMarie]

There is a Xen PV protocol for file sharing. @marmarek what do you think of it? My intuition is that it would be quite sub-optimal, as it is based on 9P, which is a network-based protocol. We should be able to do much better.

Some features I would like in a file sharing protocol:

1. Secure against malicious frontends. This is obviously table stakes for any such protocol.
2. Secure against malicious backends. User-space buffers are never exposed to the backend, avoiding time-of-check time-of-use vulnerabilities. The mount is marked in some way so that user-space filesystem indexers and other such tools know not to scan it.
3. Avoid cache-line bouncing. Each cache line should have a single writer. Multiple readers are okay.
4. Minimize data copying as much as possible. Data should be copied exactly once in the read path, from the shared grants to user-space buffers. The write path should have the backend filesystems read directly out of frontend-provided buffers. Since these buffers normally come from userspace, the backend filesystem is assumed to already treat them as untrusted and take appropriate precautions.
   • In the case of BTRFS, it may be possible for a buggy or malicious frontend to cause the backend to write incorrect checksums, resulting in reads failing with -EIO. I do not consider this to be a security problem, as the frontend can only corrupt data it already had write access to.
5. Expose most of the backend filesystem’s features. If the backend supports reflinks, subvolumes, extended attributes, or other advanced features, the frontend should be able to take advantage of them, subject to backend-side security policy.
6. Provide fine-grained access control, or at least not interfere with it. chown(), chmod(), and other such operations must work.

[imme-emosol]

Would this allow something like the suggestion mentioned on: "How to share a XEN disk with two or more virtual machines" ( https://www.suse.com/support/kb/doc/?id=000017140 )?

You need to change the option "w" (write) to w! in the XEN virtual machine configuration file located at /etc/xen/vm/VM-NAME/vm-name. Then, add the shared disk information into all virtual machines configuration files you want to share it with.

Remember that all configuration files should have the "!" (exclamation point) in the shared disk line as demonstrated below.

In the following example, xvdb disk is the one that will be shared.

• Default configuration for a non shared disk disk=[ 'file:/var/lib/xen/images/vm-name/disk0,xvda,w','file:/var/lib/xen/images/vm-name/disk1,xvdb,w', ]

than, all you need to do is including "!" as shown below: disk=[ 'file:/var/lib/xen/images/vm-name/disk0,xvda,w','file:/var/lib/xen/images/vm-name/disk1,xvdb,w!', ]

Once you have changed it, you need to refresh that information in the XEN database. You can achieve that running the following command: xm new -f vm-name

In addition, you can run "xm list -l vm-name |grep -i w!" command after changing it, in case you want to verify wether the virtual machine configuration file was updated in the xen database or not.

xm list -l vm-name |grep -i w! (mode 'w!')

In case you don't see the "!" (exclamation point), review your configuration before starting your XEN virtual machine.

[marmarek]

Would this allow something like the suggestion mentioned on: "How to share a XEN disk with two or more virtual machines" ( https://www.suse.com/support/kb/doc/?id=000017140 )?

No, this works only for filesystems designed to be mounted by multiple systems simultaneously. Most are not (especially, not ext4, xfs etc), and those that do, usually require some extra communication channel for coordination anyway. Additionally, this greatly reduces isolation between such qubes, as one qube can easily take over the other using this mechanism.

[imme-emosol]

For some qubes i do not mind to greatly reduce isolation between them.

But this is probably better suited for "set up a global share" in the qubes forum.

If shared storage goes against qubes objectives so much that it is unlikely to ever be included, perhaps there should be some sort of canonical "won't do" text about it (usable for reference in new (closable) issues).