Opt-out logs cleanup on qube removal

[emanruse]

How to file a helpful issue

Qubes OS release

4.1

Brief summary

Dom0 keeps logs for non-existing qubes.

Steps to reproduce

Create a few VMs (easiest - DispVMs), then delete them.

Expected behavior

Logs for non-existing qubes should be deleted automatically (instantly or after configurable time).

Actual behavior

Logs are preserved for infinity and pile up in /var/log/... (related subdirs)

[h01ger]

On Fri, Oct 06, 2023 at 10:29:04AM -0700, emanruse wrote:

Expected behavior

Logs for non-existing qubes should be deleted automatically (instantly or after configurable time).

indeed! this is also a privacy issue. (as well as log bloat :)

thank you for filing it!

-- cheers, Holger

⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄

Very hard to relate to those who think the first three years of the pandemic were bad because they couldn’t go to bars for a while, as opposed to because 25 million people died, 400 million were disabled, and many more continue to be unable to access public space.

[emanruse]

You are welcome.

Can you suggest a way for easy cleanup? (besides deleting files one by one which would take ages)

[kryptonik-os]

Hi, As advised by qubist in the https://forum.qubes-os.org/, I'm sharing a bug with the script he's creating in the topic https://forum.qubes-os.org/t/really-disposable-ram-based-qubes/21532 concerning a DispVMs that completely erases itself without leaving any trace (files, folders, etc...). As indicated in "Actual behavior" in your topic, there are indeed log traces in /var/log/... for all VMs created, even DispVMs. His script does a good job of erasing all traces of DipsVMs created with its script in the /var/log/ folders (which is a good thing) however, it works ONLY if the DispVM is closed BEFORE the Qubes-OS shutdow otherwise all traces remain present...

Another thing, with script, there are files left in the ~/.config/menus/applications-merged/ folder. Of course, his script is under construction:) But traces in /var/log can be deleted automatically under some conditions with this script. :)

[emanruse]

To clarify the actual additional info, @Tezeria found that not only logs of deleted qubes remain, but also files in ~/.config/menus/applications-merged/ and that is unrelated to the script discussed in the forum thread, i.e. it is purely Qubes OS issue.

[andrewdavidwong]

this is also a privacy issue.

Technically, this is only a privacy issue for Whonix disposables (if that*), since all non-Whonix qubes (including disposables) are explicitly out-of-scope regarding privacy:

https://www.qubes-os.org/faq/#what-about-privacy-in-non-whonix-qubes

Of course, it would be nice if, someday, we could also have some privacy guarantees regarding non-Whonix disposables, but that would be a separate issue (and an enhancement request rather than a bug report).

---

*(I say "if that" because it's not clear to me that Whonix disposables are intended to be amnesiac from dom0's perspective. Any opinion on this, @adrelanos?)

[emanruse]

Well, the current report is about incorrect software behavior, leaving unused files behind. Fixing it would naturally fix any additional negative effects it may cause. I hope we all agree on that.

[adrelanos]

I am not sure it would be possible, realistic to get all the local traces under control to withstand forensic analysis. Related:

• https://groups.google.com/g/qubes-devel/c/QwL5PjqPs-4
• https://github.com/QubesOS/qubes-issues/issues/904
• https://www.whonix.org/wiki/Dev/Technical_Introduction#Forensics

The related warnings, see "Amnesic Capability": https://www.whonix.org/wiki/Qubes/Disposables#Warnings

What might work and what might be more realistic is the following:

• https://github.com/QubesOS/qubes-issues/issues/4982

*(I say "if that" because it's not clear to me that Whonix disposables are intended to be amnesiac from dom0's perspective. Any opinion on this, @adrelanos?)

Would be nice to have for sure. Would be a feature unspecific to Whonix. A general Qubes feature.

[emanruse]

Just in case this has somehow been misunderstood:

This issue is about all kinds of qubes, not just disposables. Deleting any kind of qube leaves the traces described here. That has nothing to do with any goals related to forensics. It is just a bug.

If anyone thinks that certain logs should remain for some reason (e.g. for testing things), that should be an explicit option somewhere.

[adrelanos]

Logs shouldn't be removed unless it's an anti-forensics feature. To not redact any logs is the standard behavior for any *nix based operating system. Could be considered a feature, not a bug. Could be considered the expected behavior.

Even in live mode operating systems logs aren't redacted (just lost after reboot because entirely in RAM).

Debian apt remove might leave log (and config) files behind. apt purge might remove them. But that depends on each individual package, is not an anti-forensics feature and there could be oversights, for which no bugs are reported (or fixed) because of low interest in this.

Translated to Qubes, there would need to be a VM remove versus a VM purge feature. Could frame this as a missing Qubes feature. Not sure that's adding more confusion for such a minor feature than gain.

[no-usernames-left]

@adrelanos On "ordinary" operating systems I would agree, but this is Qubes.

Disposable means gone once you're done with it, like a used tissue. The logs should be too. If you think they should remain, then there should be a way to launch a disposable qube (from the CLI perhaps, to avoid cluttering up the GUI) in such a way as to retain all the logs for troubleshooting.

[adrelanos]

@adrelanos On "ordinary" operating systems I would agree, but this is Qubes.

Disposable means gone once you're done with it, like a used tissue. The logs should be too. If you think they should remain, then there should be a way to launch a disposable qube (from the CLI perhaps, to avoid cluttering up the GUI) in such a way as to retain all the logs for troubleshooting.

Why?

That is a anti-forensics for which there are existing feature requests.

My interpretation by looking at https://github.com/QubesOS/qubes-issues/issues/904 which is open since 2015 is that this is either not a high priority for Qubes and/or difficult to implement.

(And I am not criticizing this. Understandable on my part. Sure, it's a nice feature but realistically also a niche feature for which a readily available band-aid is available, that is full disk encryption. The availability of band-aids often prevents "the real solution". Anti-forensics would require a strong commitment by Qubes, speak lots of assigned developer time to that. Selective anti-forensics for some type of VMs only on an otherwise fully persistent operating system is difficult to implement. It has some of the same technically challanges which have been documented in the Kicksecure wiki article Encrypted Images.)

If the user exception is not met, if the terminology Disposable implies anti-forensics capabilities which are currently not implemented and unrealistic to materialize in the next years, then the correct way would be to rename Disposable to something more appropriate. I don't have suggestions for a better name and this can wait for feedback from the Qubes developers.

[emanruse]

Why?

Because the regular user (not a developer) simply does not need logs for non-existing VMs piling up to infinity. Having too much unnecessary data makes using the actually necessary one difficult.

That is a anti-forensics for which there are existing feature requests.

Forensics is about evidence in the context of criminal investigation. Keeping one's house (or a computer system) clean and orderly, doesn't mean one is destroying evidence of unlawful activity.

Anti-forensics was never the intent when opening this issue. The idea here is keeping up a system tidy and clean (by default). When one deletes a file, one doesn't think "I am doing anti-forensics". One just doesn't need it any more.

The purpose of logs is to keep a record of something. If one deletes a VM (or it is self-deleted, i.e. disposable), then the intent is obviously not to have a record of it. There is no promise for anti-forensics in that, that's just data management. Technically, on a magnetic hard drive, some data would still remain as regular file deletion is not a shredding operation. On an SSD, it would remain even longer due to how SSDs works. So, there is no anti-forensics expectation whatsoever.

The user may open a file (e.g. a mail attachment) in a disposable for security purposes (e.g. to protect from malware and/or network leaks). If someone may have sent an infected file, that doesn't mean the receiver is a hiding criminal.

[kryptonik-os]

I have to say that I tend to agree with the analysis of @emanruse When I did the "cleanup" in the /var/log/ folder and realized Of all the useless files that remained in memory,I say to "wow :o what is that?? :o ". Surely; It's a privacy issue, but it's mostly a A big organizational problem. For most files, they're just text files so it doesn't matter too much in terms of memory, but then, what a struggle to find one's way around! :o lol. That we keep the.log of AppVMs, why not, it can always be useful (I do not talk ,of course of the security issues, just organization. Security issues are something else again) but having ALL the.log of DispVMs I really don't see the interest...

[unman]

On Wed, Nov 01, 2023 at 07:44:00AM -0700, Tezeria wrote:

I have to say that I tend to agree with the analysis of @emanruse When I did the "cleanup" in the /var/log/ folder and realized Of all the useless files that remained in memory,I say to "wow :o what is that?? :o ". Surely; It's a privacy issue, but it's mostly a A big organizational problem. For most files, they're just text files so it doesn't matter too much in terms of memory, but then, what a struggle to find one's way around! :o lol. That we keep the.log of AppVMs, why not, it can always be useful (I do not talk ,of course of the security issues, just organization. Security issues are something else again) but having ALL the.log of DispVMs I really don't see the interest...

There are many cases where working in a disposable, and the disposable closes unexpectedly, it is useful to have the logs available. Useful perhaps not for the average user, but useful for those who might want to help the average user.

I see the issue as no different from any other system logs. If this is a concern have the logs rotated and culled.

[emanruse]

Please note that the current issue is not only about disposables.

The actual cases when a user provides requested debugging info to an expert are quite rare compared to what the average user normally does. Even though I agree that may be useful, it is not a valid reason to force everyone to keep hundreds of logs for every non-existing qube for months.

Some desktop distros keep journal files on tmpfs, so they don't survive reboot. Perhaps we can have something similar for disposables (rather than wear our SSDs with unnecessary writes of mostly unnecessary data)?

If this is a concern have the logs rotated and culled.

Looking at libvirtd's logrotate config, for example, it seems logs are kept for 4 weeks before rotating and rotates happen only for files > 100k (which seems quite far below the logs of disposables). Maybe this is a separate issue though.

[emanruse]

(along the lines of anti-forensincs which was mentioned) Something else I noticed:

After stopping and deleting a qube, a "non-file" with 0 bytes size remains:

/run/qubes/audio-control.=

It cannot be removed even as root, it does not appear in lsof, so I have no idea what is using/locking it.

Another observation: renaming a qube does not rename its logs. Those files remain with their old names and new ones are created. This may probably result in confusion in cases such as:

A -> renamed to -> B C -> renamed to -> A (and appending log data in A's ex log files)

[marmarek]

I think a good compromise would be opt-out logs cleanup on qube removal. Like, a cleanup_logs propery on a qube, default to true. Then remove_from_disk function in core-admin will check it and if set to True, remove qube-related logs. If you want to preserve them, set it to false (for DispVMs, you'd set it on DispVM template, properties inheritance will do the trick).

Note it still won't erase all traces of existence of a qube, for example a few global logs (like system journal) will still have mentions of that qube. But I think this part clear here.

[emanruse]

I like your proposal. It is still good to have (an option for) disposable's logs on tmpfs though.

Is it possible to also have automatic renaming of logs when renaming a qube?

[kryptonik-os]

@emanruse

After stopping and deleting a qube, a "non-file" with 0 bytes size remains: /run/qubes/audio-control.= It cannot be removed even as root, it does not appear in lsof, so I have no idea what is using/locking it.

I've noticed this but /run/qubes/audio-control.<qube-name>= , after the QUBES-OS reboot, that goes away (for me)

[emanruse]

Everything on all tmpfs mounts (/run included) goes away on reboot. The question is what keeps using that audio-control file after a qube is destroyed.

[kryptonik-os]

Everything on all tmpfs mounts (/run included) goes away on reboot. My bad :)

Like it disappear at reboot, i never ask me this question...

[marmarek]

On Wed, Nov 15, 2023 at 05:51:44AM -0800, Tezeria wrote:

@emanruse

After stopping and deleting a qube, a "non-file" with 0 bytes size remains: /run/qubes/audio-control.=

The "=" is not part of the file name, it's how ls shows you it is a socket.

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab

[emanruse]

Great tip. Thank you!

[emanruse]

In 4.2, when a qube is deleted, it also leaves behind a .menu file:

@.***:~ > qvm-create personal-dvm --label red

@.**:~ > find ~ -regextype posix-egrep -regex '.personal(.)dvm.' /home/user/.local/share/desktop-directories/qubes-vm-directory_personal_ddvm.directory /home/user/.local/share/qubes-appmenus/personal-dvm /home/user/.local/share/qubes-appmenus/personal-dvm/apps.icons /home/user/.local/share/qubes-appmenus/personal-dvm/apps.icons/firefox.png /home/user/.local/share/qubes-appmenus/personal-dvm/apps /home/user/.local/share/qubes-appmenus/personal-dvm/apps/qubes-vm-directory_personal_ddvm.directory /home/user/.local/share/qubes-appmenus/personal-dvm/apps/org.qubes-os.qubes-vm-settings._personal_ddvm.desktop /home/user/.local/share/qubes-appmenus/personal-dvm/apps/org.qubes-os.vm._personal_ddvm.firefox.desktop /home/user/.local/share/applications/org.qubes-os.qubes-vm-settings._personal_ddvm.desktop /home/user/.local/share/applications/org.qubes-os.vm._personal_ddvm.firefox.desktop /home/user/.gnome/apps/org.qubes-os.qubes-vm-settings._personal_ddvm.desktop /home/user/.gnome/apps/org.qubes-os.vm._personal_ddvm.firefox.desktop /home/user/.config/menus/applications-merged/user-qubes-vm-directory_personal_ddvm.menu

@.***:~ > qvm-remove personal-dvm This will completely remove the selected VM(s)... personal-dvm Are you sure? [y/N] y

@.**:~ > find ~ -regextype posix-egrep -regex '.personal(.)dvm.' /home/user/.config/menus/applications-merged/user-qubes-vm-directory_personal_ddvm.menu

[emanruse]

Something more:

After upgrading to 4.2, now I notice even duplicated .menu files for (almost) all qubes my qubes with slightly different names - one is with hyphen, the other is with underscore:

@.***:~/.config/menus/applications-merged > ls -1 ... user-qubes-vm-directory-TEST.menu user-qubes-vm-directory_TEST.menu ...