Whonix DVM closes before loading app with 4.14.103 kernel

farrilis commented 5 years ago

Qubes OS version:

R4.0

Affected component(s) or functionality:

whonix-ws-14-dvm, Tor Browser or Konsole only when using kernel 4.14.103-1

Steps to reproduce the behavior:

Open an app from the sub-menu whonix-ws-14-dvm in the main Qubes applications menu

Expected or desired behavior:

Qube boots, application opens, qube remains booted

Actual behavior:

Once a few (around 4-5) other VM's are loaded, anything from whonix-ws-14-dvm boots, then goes directly into shutdown after booting. App does not load.

Only when using kernel 4.14.103-1, setting the previous kernel (4.14.74) works around the issue

General notes:

The same issue does not happen with debian-9-dvm in the same conditions (>4-5 AppVMs open, kernel 4.14.103 in use by the debian-9 DVM)

I have consulted the following relevant documentation:

I am aware of the following related, non-duplicate issues:

marmarek commented 5 years ago

Can you check console output of such DispVM? It should be in dom0 in /var/log/xen/console/guest-dispXXX.log (the actual VM name you can get from startup notification).

farrilis commented 5 years ago

All lines in the log status are 'OK', except here:

Starting udev Kernel Device Manager... [.[[0;32m OK .[0m] Reached target Local Files Systems (Pre). [.[[0;1;31m FAILED .[0m] Failed to start Load Kernel Modules. See 'systemctl status systemd-modules-load.service' for more details

systemctl status systemd-modules-load.service and journalctl -u systemd-modules-load.service tell me about what happened many hours ago, nothing from the time when I opened the whonix-ws-14-dvm

The last lines of /var/log/xen/console/guest-dispXXX.log are:

Starting Daily apt upgrade and clean activities... [.[[0;32m OK .[0m] Started LSB: Anonymous remailer client and server. [.[[0;32m OK .[0m] Started Daily APT upgrade and clean activities. [.[[0;32m OK .[0m] Started Qubes misc post-boot actions. [.[[0;32m OK .[0m] Started User Manager for UID 1000. [.[[0;32m OK .[0m] Started Secure Distributed Web Date. [.[[0;32m OK .[0m] Started redirect /var/run/tor/control to Whonix-Gateway port 9051. [.[[0;32m OK .[0m]

marmarek commented 5 years ago

On Tue, Mar 26, 2019 at 11:47:57AM -0700, farrilis wrote:

All lines in the log status are 'OK', except here:

Starting udev Kernel Device Manager... [.[[0;32m OK .[0m] Reached target Local Files Systems (Pre). [.[[0;1;31m FAILED .[0m] Failed to start Load Kernel Modules. See 'systemctl status systemd-modules-load.service' for more details

systemctl status systemd-modules-load.service and journalctl -u systemd-modules-load.service tell me about what happened many hours ago, nothing from the time when I opened the whonix-ws-14-dvm

This should be already fixed by https://github.com/QubesOS/updates-status/issues/874 See also https://github.com/QubesOS/qubes-issues/issues/4608 Anyway, should be harmless and definitely should not result in DispVM closed prematurely.

The last lines of /var/log/xen/console/guest-dispXXX.log are:

Nothing out of ordinary. Can you check if the same issue applies to anon-whonix (Whonix Workstation, but not DispVM)?

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

farrilis commented 5 years ago

Nothing out of ordinary. Can you check if the same issue applies to anon-whonix (Whonix Workstation, but not DispVM)?

anon-whonix boots and stays up.

I checked the anon-whonix logs at /var/log/xen/console/guest-anon-whonix.log (the most recent logfile), there are additional error lines that are not in the guest-disp log.

marmarek commented 5 years ago

anon-whonix boots and stays up.

Does the application show up (Tor Browser, or whatever you've tried to start there)?

I checked the anon-whonix logs at /var/log/xen/console/guest-anon-whonix.log (the most recent logfile), there are additional error lines that are not in the guest-disp log.

What exactly?

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

farrilis commented 5 years ago

Does the application show up (Tor Browser, or whatever you've tried to start there)?

No, notification for VM started is quickly followed (1-2s) by notification for VM halted, no application window appears

What exactly?

switch_root: failed to mount moving /dev to /sysroot/dev: Invalid argument switch_root: Forcing unmount of /dev switch_root: failed to mount moving /proc to /sysroot/proc: Invalid argument switch_root: Forcing unmount of /proc switch_root: failed to mount moving /sys to /sysroot/sys: Invalid argument switch_root: Forcing unmount of /sys switch_root: failed to mount moving /run to /sysroot/run: Invalid argument switch_root: Forcing unmount of /run

and also:

FAILED Failed umounting /lib/modules/4.14.103-1.pvops.qubes.x86_64...

marmarek commented 5 years ago

On Tue, Mar 26, 2019 at 08:24:11PM +0000, farrilis wrote:

Does the application show up (Tor Browser, or whatever you've tried to start there)?

No, notification for VM started is quickly followed (1-2s) by notification for VM halted, no application window appears

anon-whonix?

What exactly?

switch_root: failed to mount moving /dev to /sysroot/dev: Invalid argument switch_root: Forcing unmount of /dev switch_root: failed to mount moving /proc to /sysroot/proc: Invalid argument switch_root: Forcing unmount of /proc switch_root: failed to mount moving /sys to /sysroot/sys: Invalid argument switch_root: Forcing unmount of /sys switch_root: failed to mount moving /run to /sysroot/run: Invalid argument switch_root: Forcing unmount of /run

and also:

FAILED Failed umounting /lib/modules/4.14.103-1.pvops.qubes.x86_64...

Those should not cause the issue you're experiencing.

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

farrilis commented 5 years ago

anon-whonix?

it boots, applications work, everything functions as expected using kernel 4.14.103-1

farrilis commented 5 years ago

I tried 4.19.29-1 from current-testing, same apparent issue: whonix-ws-14-dvm boots, doesn't open the application, then shuts down. debian-9-dvm using the same kernel is fine however.

However, I was possibly wrong to say that 4.14.103 always fails with whonix-ws-14-dvm, as several attempts using 4.19.29-1 has proved the issue is intermittent. Trying again with 4.14.103 has demonstrated the same behaviour; whonix-ws-14-dvm fails frequently, but not always. This might be true of debian-9-dvm too, but so far it has never failed (I may need to change the title of this issue)

farrilis commented 5 years ago

Tried dozens of times to get debian-9-dvm using kernel 4.14.103 to fail the same way, it hasn't happened once. Tried the simplest application I could think of (xterm) in whonix-ws-14-dvm, it behaves the same as Tor Browser or any other application (~50% of attempts, the VM shuts down after loading, no application window appears)

mig5 commented 5 years ago

I am having the same problem. It also happens if I try the older 4.14.74-1 kernel.

There are no errors for me in the /var/log/xen/console/guest-dispXXXX.log, it lands at the Whonix login prompt. But the app I wanted to open in the dispVM never opens, and after 30-45 seconds the dispVM automatically shuts down.

A regular fedora29 dispVM works just fine.

mig5 commented 5 years ago

Possible useful debug with a traceback from /var/log/qubes/qubes.log

disp.txt

brendanhoar commented 5 years ago

I haven't looked deeply into the issue, but I've seen something similar. I'm using -testing and the most recent kernel available.

If, on my test box, I start up several disposable whonix-ws VMs in parallel (opening a console), at least half shutdown w/o a window appearing. If I open them up one by one and wait for the console window to appear before opening the next one, the completion rate is much higher.

marmarek commented 5 years ago

But the app I wanted to open in the dispVM never opens, and after 30-45 seconds the dispVM automatically shuts down.

Are you sure it was 30-45 seconds? There is a 60 seconds timeout involved - see qvm-prefs whonix-ws-14-dvm qrexec_timeout.

mig5 commented 5 years ago

Hi @marmarek, you are right. I raised the timeout to 280s (arbitrarily). Timing it just now from the moment I click on Tor Browser in the Qubes menu of a Whonix DVM, it took about 93 seconds before it loaded. But it does load now. Thanks

marmarek commented 5 years ago

That's quite a long time. Can you check /var/log/xen/console/guest-dispXXXX.log (put the actual dispvm number there) what took it so long?

brendanhoar commented 5 years ago

EDIT: nevermind, set timeout to 360 seconds and some still failed to start.

@mig5 - out of curiosity, what are the five largest number of seconds shown in this result set:

# last timestamp shown before console shows login screen.

grep "fbcon: Taking over console" `grep whonix -l /var/log/xen/console/*disp*`|grep -v "\:\[    "|more

I will compare it to the numbers I get when I start multiple disposable whonix instances and some fail to stay running.

Brendan

mig5 commented 5 years ago

@brendanhoar looks like between 10 and 18 seconds

guest-disp1270.log-20190520:[   10.930697] fbcon: Taking over console
guest-disp1345.log-20190624:[   13.235786] fbcon: Taking over console
guest-disp152.log-20190520:[   17.855179] fbcon: Taking over console
guest-disp1686.log-20190526:[   12.130168] fbcon: Taking over console
guest-disp1802.log-20190611:[   18.722486] fbcon: Taking over console

@marmarek I believe I have found the issue.

I watched the console log as I booted a Whonix dispVM just now.

It took only 27s to get to the console prompt after booting, Nothing else appears in the log after that. But it took til 1m38s for Tor Browser to start. In other words, an additional 1m10s from the completed boot/at console before the GUI app loaded.

In the syslog of the Whonix dispVM I see a lot of activity by a task 'first-boot-home-population' which seems to be copying a large amount of tor browser data from /var/cache/tb-binary/ into the user's /home/user/.cache and .tb directories. This process appears to take just over a minute (timestamp in log of the first-boot-home-population process): 1:48:31 - 1:49:38

After this process completes, the remaining startup processes complete and Tor Browser loads:

Jun 26 01:48:30 host first-boot-home-population[766]: + user_name=user
Jun 26 01:48:30 host first-boot-home-population[766]: + home_dir=/home/user
Jun 26 01:48:30 host first-boot-home-population[766]: + cache_folder=/home/user/.tb
Jun 26 01:48:30 host first-boot-home-population[766]: + done_file=/home/user/.tb/first-boot-home-population.done
Jun 26 01:48:30 host first-boot-home-population[766]: + '[' -e /home/user/.tb/first-boot-home-population.done ']'
Jun 26 01:48:30 host first-boot-home-population[766]: + command -v qubesdb-read
Jun 26 01:48:30 host first-boot-home-population[766]: ++ qubesdb-read /qubes-vm-type
Jun 26 01:48:30 host first-boot-home-population[766]: + qubes_vm_type=AppVM
Jun 26 01:48:30 host first-boot-home-population[766]: ++ qubesdb-read /name
Jun 26 01:48:30 host first-boot-home-population[766]: + qubes_vm_name=disp2816
Jun 26 01:48:30 host first-boot-home-population[766]: + grep -q '\-dvm'
Jun 26 01:48:30 host first-boot-home-population[766]: + echo disp2816
Jun 26 01:48:30 host first-boot-home-population[766]: + '[' AppVM = TemplateVM ']'
Jun 26 01:48:30 host first-boot-home-population[766]: + '[' '!' -d /home/user ']'
Jun 26 01:48:30 host first-boot-home-population[766]: + '[' -d /var/cache/tb-binary ']'
Jun 26 01:48:30 host misc-post.sh[686]: Failed to read /qubes-gateway6
Jun 26 01:48:30 host misc-post.sh[686]: Cannot get device udp-fragmentation-offload settings: Operation not supported
Jun 26 01:48:30 host misc-post.sh[686]: Cannot get device udp-fragmentation-offload settings: Operation not supported
Jun 26 01:48:30 host misc-post.sh[686]: Cannot get device udp-fragmentation-offload settings: Operation not supported
Jun 26 01:48:30 host misc-post.sh[686]: Cannot get device udp-fragmentation-offload settings: Operation not supported
Jun 26 01:48:30 host misc-post.sh[686]: Could not change any device features
Jun 26 01:48:30 host first-boot-home-population[766]: + chown --recursive user:user /var/cache/tb-binary
Jun 26 01:48:30 host systemd[1]: Started anon-ws-disable-stacked-tor.
Jun 26 01:48:30 host misc-post.sh[686]: SIOCADDRT: File exists
Jun 26 01:48:30 host misc-post.sh[686]: SIOCADDRT: File exists
Jun 26 01:48:30 host systemd[1]: Started Qubes misc post-boot actions.
Jun 26 01:48:31 host socat-unix-sockets[813]: INFO: No processes launched into background, ok.
Jun 26 01:48:31 host systemd[1]: Started Whonix legacy versions fixes.
Jun 26 01:48:31 host systemd[1]: Started whonixcheck.
Jun 26 01:48:31 host systemd[1]: Started Secure Distributed Web Date.
Jun 26 01:48:31 host sdwdate[771]: 2019-06-26 01:48:31 - sdwdate - INFO - sdwdate started. PID: 771
Jun 26 01:48:31 host sdwdate[771]: 2019-06-26 01:48:31 - sdwdate - INFO - create temp_dir: /tmp/tmp.4MyTxIpDop
Jun 26 01:48:31 host sdwdate[771]: 2019-06-26 01:48:31 - sdwdate - INFO - Tor socks host: 10.137.0.7 Tor socks port: 9108
Jun 26 01:48:31 host sdwdate[771]: 2019-06-26 01:48:31 - sdwdate - INFO - Running sdwdate main loop. iteration: 1 / 10000
Jun 26 01:48:31 host first-boot-home-population[766]: + shopt -s dotglob
Jun 26 01:48:31 host first-boot-home-population[766]: + shopt -s nullglob
Jun 26 01:48:31 host first-boot-home-population[766]: + for folder_name in /var/cache/tb-binary/*
Jun 26 01:48:31 host first-boot-home-population[766]: + sudo --non-interactive -u user cp --verbose --recursive --no-clobber /var/cache/tb-binary/.cache /home/user
Jun 26 01:48:31 host first-boot-home-population[766]: '/var/cache/tb-binary/.cache' -> '/home/user/.cache'
Jun 26 01:48:31 host first-boot-home-population[766]: '/var/cache/tb-binary/.cache/tb' -> '/home/user/.cache/tb'
[....snip lots of copying of more data from first-boot-home-population process here ...]
Jun 26 01:49:38 host systemd[1]: Started Copy Tor Browser from /var/cache/tb-binary to user home at First Boot Service.
Jun 26 01:49:38 host systemd[1]: Starting Qubes GUI Agent...
Jun 26 01:49:38 host systemd[1]: Started Qubes GUI Agent.
Jun 26 01:49:38 host systemd[1]: Reached target Multi-User System.
Jun 26 01:49:38 host systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jun 26 01:49:38 host systemd[1]: Started Update UTMP about System Runlevel Changes.
Jun 26 01:49:38 host systemd[1]: Startup finished in 5.348s (kernel) + 1min 16.716s (userspace) = 1min 22.064s.
Jun 26 01:49:38 host qubes-gui[2287]: Waiting on /var/run/xf86-qubes-socket socket...
Jun 26 01:49:38 host systemd[1]: Started Session c3 of user user.
Jun 26 01:49:38 host qubes-gui[2287]: Ok, somebody connected.
Jun 26 01:49:38 host qrexec-agent[637]: send exit code 0
Jun 26 01:49:38 host qrexec-agent[637]: pid 712 exited with 0
Jun 26 01:49:38 host qrexec-agent[637]: eintr
Jun 26 01:49:39 host qrexec-agent[637]: eintr
Jun 26 01:49:39 host pulseaudio[2455]: vchan module loading
Jun 26 01:49:39 host pulseaudio[2455]: play libvchan_fd_for_select=19, ctrl=0x59f80f6cbf20
Jun 26 01:49:39 host pulseaudio[2455]: rec libvchan_fd_for_select=23, ctrl=0x59f80f6cc030
Jun 26 01:49:40 host pulseaudio[2455]: sink cork req state =1, now state=-2
Jun 26 01:49:40 host pulseaudio[2455]: source cork req state =1, now state=-2
Jun 26 01:49:40 host pulseaudio[2455]: Failed to open cookie file '/home/user/.config/pulse/cookie': No such file or directory
Jun 26 01:49:40 host pulseaudio[2455]: Failed to load authentication key '/home/user/.config/pulse/cookie': No such file or directory
Jun 26 01:49:40 host pulseaudio[2455]: Failed to open cookie file '/home/user/.pulse-cookie': No such file or directory
Jun 26 01:49:40 host pulseaudio[2455]: Failed to load authentication key '/home/user/.pulse-cookie': No such file or directory
Jun 26 01:49:46 host pulseaudio[2455]: source cork req state =2, now state=1
Jun 26 01:49:46 host pulseaudio[2455]: sink cork req state =2, now state=1
Jun 26 01:49:47 host systemd[1]: Started redirect /var/run/anon-ws-disable-stacked-tor/127.0.0.1_9151.sock to Whonix-Gateway port 9051.
Jun 26 01:49:47 host systemd[1]: Started redirect /var/run/anon-ws-disable-stacked-tor/127.0.0.1_9150.sock to Whonix-Gateway port 9150.
[....snip...]
Jun 26 01:49:52 host systemd[1]: Started sets up Qubes-Whonix-Gateway external and internal network interface.
Jun 26 01:50:52 host systemd[1]: Started Session c4 of user user.
Jun 26 01:50:52 host qrexec-agent[637]: eintr
Jun 26 01:50:58 host mate-notificati[3616]: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: Recreating ksycoca file ("/home/user/.cache/ksycoca5_en_DUcAY8MHK9CqUalxnSyf45tD93g=", version 303)
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: Menu "applications-kmenuedit.menu" not found.
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: Saving
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: org.kde.kurifilter-ikws: "Jun 26 01:48:31 host first-boot-home-population[766]: + shopt -s dotglob"
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: org.kde.kurifilter-ikws: Keywords Engine: Loading config...
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: org.kde.kurifilter-ikws: Web Shortcuts Enabled:  true
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: org.kde.kurifilter-ikws: Default Shortcut:  ""
Jun 26 01:51:32 host qubes.StartApp+qubes-run-terminal-dom0: org.kde.kurifilter-ikws: Keyword Delimiter:  :
Jun 26 01:52:53 host systemd[1]: Starting Qubes check for VM updates and notify dom0...
Jun 26 01:52:54 host systemd[1]: Started redirect 127.0.0.1:9050 to Whonix-Gateway port 9050.
Jun 26 01:53:16 host systemd[1]: Started Qubes check for VM updates and notify dom0.

Thus we can probably say that it is the massive I/O of this 'first-boot-home-population' process copying all the Tor Browser cache data into the user's home directory, which blocks the remainder of the startup, and potentially then trips the qrexec timeout.

Probably it depends on the speed of the users' disk etc as to how long this can take.

marmarek commented 5 years ago

This probably could be mitigated by starting whonix-ws-dvm once, so the process completes there and persists.

On one hand, copying Tor Browser from /var/cache/tb-browser at DispVM start time takes time, but on the other hand it will get you newer version (if it's updated in template) - instead of older version cached in /home (if present). @adrelanos what should be the best solution here? Or maybe the copy operation could be optimized somehow? For example only symlink to files which aren't modified (binaries, libraries etc), instead of copying - if feasible...

brendanhoar commented 5 years ago

Do you have an SSD or a hard drive?

I've noticed that several versions of both the torbrowser and/or i2pbrowser application tree, as well as the downloaded installation archives, seem to accumulate in the template's /var/cache tree some of which is moved to /home/user/{.tb,.i2pb} during startup of the disposable VM.*

So...can you scan the template to find out:

a) the size of /var/cache in MB (e.g. sudo du --si /var/cache) b) how many versions of the torbrowser and/or i2pbrowser installations are cached in the /var/cache tree? c) how many version of the torbrowser and/or i2pbroser archives are cached in the /var/cache tree?

My /var/cache tree is currently ~1500MB, and that's after removing several stale {tor,i2p} browser installs from /var/cache/tb-binary/{.tb,.i2pb} as well as several installer archives of {tor,i2p} browser from /var/cache/.cache/{tb,i2pb}/files/ ...

EDIT: so I just removed the old versions of both and now the template's /var/cache is ~ 1100MB.

In the past, I've removed several installs of each browser from /var/cache in a single template cleanup session, as well a couple copies of the downloaded archives during the same session.

I wouldn't be surprised if the issue isn't so much startup timeout, but the significant IO causing vchan or some other expected conduit for queries/responses to stall, after which dom0 determines the VM isn't working properly and forces it to die. The larger the /var/cache (and/or the less responsive the storage layer), the more likely that dom0 is to terminate the VM.

Or at least, that's my theory.

Brendan

* I realized this when updating torbrowser in the disposableVM once and had a terminal open to the /home/user/.tb/tor-browser subdir with some configuration information in it, only to find that I was now in a different tree (as my tree had been renamed, and the updated version was now /home/user/.tb/tor-browser). I looked in .tb and there were THREE older versions of torbrowser (with .old-datetimestamp suffix) as well as the now-current one.

mig5 commented 5 years ago

I had about 822MB of /var/cache. Indeed I had a couple of torbrowser.old.xxxx-xxxx dirs in /var/cache/tb-binary/.tb , in the Template.

After removing the old ones in the Template, my Whonix dispVM now spins up in about 35 seconds. A dramatic improvement

andrewdavidwong commented 5 years ago

I looked in .tb and there were THREE older versions of torbrowser (with .old-datetimestamp suffix) as well as the now-current one.

I remember seeing somewhere that "Tor Browser Downloader by Whonix ™" intentionally keeps these three backups. I can't find it in the documentation, so maybe I saw that message in the tool itself.

CC: @adrelanos

adrelanos commented 5 years ago

I was thinking it is wasteful anyhow to copy Tor Browser from root image to home image for every VM. Even those who might never use it. Perhaps only do the copy when actually starting Tor Browser?

technical background of all of this mess:

https://www.whonix.org/wiki/Tor_Browser/Advanced_Users#Tor_Browser_Update:_Technical_Details

Marek Marczykowski-Górecki:

@adrelanos what should be the best solution here?

Someone solve the root cause of this issue: non-existing deb package.

https://trac.torproject.org/projects/tor/ticket/3994

https://trac.torproject.org/projects/tor/ticket/5236

On one hand, copying Tor Browser from /var/cache/tb-browser at DispVM start time takes time, but on the other hand it will get you newer version (if it's updated in template) - instead of older version cached in /home (if present).

Indeed.

@adrelanos what should be the best solution here? Or maybe the copy operation could be optimized somehow? For example only symlink to files which aren't modified (binaries, libraries etc), instead of copying - if feasible...

I don't think melding in the folder itself is feasible. Too dynamic project to keep up.

Andrew David Wong:

I looked in .tb and there were THREE older versions of torbrowser (with .old-datetimestamp suffix) as well as the now-current one.

I remember seeing somewhere that "Tor Browser Downloader by Whonix ™" intentionally keeps these three backups. I can't find it in the documentation, so maybe I saw that message in the tool itself.

TB_KEEP_OLD_VERSIONS_COUNT (defaults to 3) was implemented by Marek a while ago.

ticket: https://phabricator.whonix.org/T671

I am happy to set TB_KEEP_OLD_VERSIONS_COUNT to 0 by default at expense to those who customized that folder and then loose their settings which hopefully is a rare case but I am sure we are going to hear from them?

brendanhoar commented 5 years ago

I was thinking it is wasteful anyhow to copy Tor Browser from root image to home image for every VM. Even those who might never use it. Perhaps only do the copy when actually starting Tor Browser?

Yes, this trade-off seems worth it. Folks who use whonix VMs, particularly disposable whonix VMs, use them for multiple purposes, not just TB (or I2PB). Disposable VMs that use TB really don't need the older copies (ever), and disposable VMs that won't be used for TB don't need TB moved to /home/user/.tb during startup (e.g. using a disposable whonix VM via "open in VM").

An additional mitigation would be to have the TB/I2PB startup script determine whether it is running a disposable VM and if so, only copy the most recent TB/I2PB directory to home from cache? e.g.

if [ ${dispVM} = true ] ; then mkdir /home/user/.tb cp -R /var/cache/tb-binary/.tb/tor-browser /home/users/.tb/ else mkdir /home/user/.tb cp -R /var/cache/tb-binary/.tb/* /home/users/.tb/ fi

B

adrelanos commented 5 years ago

As per tb-updater git master: No more backups will be kept by default.

adrelanos commented 5 years ago

This is now in Whonix 15 / developers repository.

tb-updater / tb-starter has been modified to no longer copy Tor Browser by default to user home at first boot. Instead it will be done at the actual first start of Tor Browser.

This is also already available Whonix 15 / developers repository.

It's not functional yet, possibly due to a Qubes issue. Need some help here.

When one starts xfce4-terminal in a Whonix 15 DispVM, there is no ~/.tb folder as expected.
When one starts xfce4-terminal in a Whonix 15 DispVM, and then starts torbrowser on the command line, that is functional as expected.
When one starts Tor Browser in a Whonix 15 DispVM from the Qubes start menu, a DispVM will start and terminate after approximate 20 seconds.

qvm-prefs whonix-ws-14-dvm qrexec_timeout is 60.

What's the problem here?

adrelanos commented 5 years ago

Might actually be unrelated.

curlpit could be this:

https://github.com/Whonix/tb-starter/commit/11ed26d14ed308891db5fc366a1002da41bdd3c1

adrelanos commented 5 years ago

Was indeed a different issue which is now being worked around. (https://github.com/Whonix/tb-starter/commit/bd9f6f8c991ad0c170e63ce1412006239aee0760) (https://phabricator.whonix.org/T818) Issue reported here: Qubes start menu incompatible with DispVMs launching GUI applications into the background https://github.com/QubesOS/qubes-issues/issues/5129

adrelanos commented 5 years ago

tb-updater / tb-starter has been modified to no longer copy Tor Browser by default to user home at first boot. Instead it will be done at the actual first start of Tor Browser. This is now functional. This is available Whonix 15 / developers repository.

adrelanos commented 5 years ago

This should be enough to fix the original issue.

adrelanos commented 5 years ago

Using an SSD it still takes ~ 30 seconds to run Tor Browser.

Running tb-updater in DVM template is prevented by default but can be optionally re-enabled, it's documented as per below.

For simplicity of documentation and usability it was decided that tb-updater should only be run TemplateVM automatically due to package upgrade; in TemplateVM or manually; or manually in AppVM. But not in DVM Template.

marmarek commented 5 years ago

Crazy (and hacky...) idea: since neither private or root volume is persistent in DisposableVM, maybe TorBrowser could be run from /var/cache directly (if not present in ~/.tb before)? I mean, create a symlink and possibly adjust permissions when starting in DispVM, instead of copying. That would be bad for non-DisposableVM, since changes in root volume are lost (unless StandaloneVM/TemplateVM), but in DisposableVM it should be fine.

adrelanos commented 5 years ago

This is now implemented in Whonix 15 developers repository.

adrelanos commented 5 years ago

The bad thing about running directly from /var/cache/tb-binary directly is that it breaks mandatory access control framework firejail.

mount --bind might work better? Any reasons against that?

marmarek commented 5 years ago

mount --bind in dispvm should be fine

brendanhoar commented 5 years ago

Not a criticism of the solution, just wanted to be sure I know the impact:

This change does mean that in disposable whonix VMs, all torbrowser writes (non-RAM cache, etc) will be on the volatile rw based on the root filesystem, not in the (also-volatile) private volume, right? If tb or i2pb end up needing more space to run, the template's system/root volume max size would need to be increased, right?

marmarek commented 5 years ago

This change is just about tb, not i2pb (unless it shares .tb directory). Yes, exactly, root volume would need to be extended. In the current implementation (which may change), it should be even possible to do that on the DisposableVM itself (using cmdline, because GUI do not allow that), not its template, if you want it just one time.

brendanhoar commented 5 years ago

I2pb is handled just like tb in Whonix 14: copied over to private volume on each disposable vm start.

adrelanos commented 5 years ago

Don't ask me about i2pb - I don't maintain, review that code besides non-maliciousness.

adrelanos commented 5 years ago

Using an SSD it still takes ~ 30 seconds to run Tor Browser.

This was reduced to ~ 15 seconds.

Takes 10 seconds to get notification in Qubes dom0 that VM was started. Another 15 seconds to see Tor Browser.

Using bind mount now in Qubes DispVM to avoid copying Tor Browser to user home folder.

This is available in all Whonix repositories.

This hopefully fixed the qrexec timeout issues.

adrelanos commented 4 years ago

This was reduced to ~ 15 seconds.

This is now the case in Whonix stable repository.

This is also the case in these new Qubes-Whonix templates which are currently being tested (but announcement not written yet) (and which might become the new stable version shouldn't there be any other release blockers):

adrelanos commented 2 years ago

Closable?

DemiMarie commented 2 years ago

I think so, yes.

QubesOS / qubes-issues