Keyboard layout of restored qubes stuck

[UndeadDevel]

I was going to report this first on the Qubes issue tracker, but it does seem to be wyng specific (wyng and wyng-util-qubes are on the main branches; for wyng: prog_date = "20240515"; the util file was updated the same day):

I had noticed for a few days that there were problems with switching the keyboard layout in some qubes and today investigated: the keyboard layout will not change from its default in all qubes that were restored from wyng, with the exception of the gpg-vault qube, which was restored with an error (wyng-util-qubes crashed, reported in #25). VMs restored from my regular Qubes backup (qvm-backup) and VMs not restored at all have no such issue. Deleting an affected qube and restoring with wyng-util-qubes again does not fix the problem; backing up the qube with the regular qubes tools, deleting it and restoring it with the regular qubes tools does not fix the issue; only restoring from my last big regular qvm-backup backup fixes the issue. I didn't have any keyboard layout issues like this ever since I upgraded to QubesOS 4.2, so for >6 months and, as stated, only those qubes which I restored with wyng have the issue, so it's very likely due to wyng (or wyng-util-qubes).

So I'm likely going to have to create new qubes and manually move the user data from the affected ones over unless you have an idea of how to fix this...can wyng tell me which files are different between two sessions?

[tasket]

I tried to reproduce the problem by setting a vm's keyboard_layout pref to something other than the system default. Then I used wyng-util-qubes backup vmname to back it up. Next I deleted the vm from the system with qvm-remove and finally I restored it. The vm with correct keyboard_layout setting was restored without errors.

I suggest trying a restore and checking that pref with qvm-prefs both before and after the restore.

In the worst case, if the settings aren't restored, you should be able to set the keyboard pref manually with qvm-prefs instead of having to create new vms and then moving the data.

In the past there have been problems with Qubes prefs not being saved unless the pref was set twice with a delay in between; in that case Qubes was not responding correctly to changes in settings. As with the spot test you did with qvm-backup, even the Qubes-supplied tools might not handle vm settings correctly.

[tasket]

BTW, you can use --debug on the util command line and when it gets to the prefs setting part, each pref and its value will be displayed.

[UndeadDevel]

I don't think that it's connected to the keyboard_layout property necessarily as it's the same for affected and non-affected qubes.

I did some more testing:

1. backup with the regular qubes tools of a non-affected qube (never restored with wyng) that is based on the same template as an affected qube, deleting that qube and then restoring it with the regular qubes tools -> layout switching still works.
2. Deleting that qube again and this time restoring with wyng -> layout switching broken
3. Deleting it again and restoring again from backup of step 1 with qubes tools -> layout switching works again

So it seems the restoration with wyng breaks something that is then "saved" somehow and will affect regular qubes backup restoration; if a wyng restore was not done on the qube then it won't break upon backup and restore with the regular qubes tools. All throughout that test the qvm-prefs keyboard_layout value didn't change. Output from step 2:

[user@dom0 ~]$ sudo wyng-util-qubes --dest=qubes://path/to/backup/laptop.backup --debug restore testqube
wyng-util-qubes v0.9beta rel 20240515
['/usr/local/bin/wyng', '--dest=qubes://path/to/backup/laptop.backup', '--json', 'list']
['/usr/local/bin/wyng', '--dest=qubes://path/to/backup/laptop.backup', '-u', '--quiet', '--save-to=/tmp/wuqzg41jqo_/qmeta.tgz', '--session=20240522-122552', 'receive', 'wyng-qubes-metadata']
['tar', '-xzf', '/tmp/wuqzg41jqo_/qmeta.tgz']

VMs matched in session 20240522-122552:
 testqube
Warning:  Restoring to existing VMs will overwrite them.
Continue [y/N]? y

Restoring VM data volumes:
JSON:
 {"qubes_dom0/vm-pool": [["vm-testqube-private", "vm-testqube-private"]]}
['/usr/local/bin/wyng', '--dest=qubes://path/to/backup/laptop.backup', '-u', '--sparse-write', '--local-from=/tmp/wuqzg41jqo_/wuq_vols.lst', '--session=20240522-122552', 'receive']
Wyng 0.8beta release 20240515
Encrypted archive 'qubes://path/to/backup/laptop.backup' 
Last updated 2024-05-22 14:10:13.157421 (+01:00)

Receiving volume 'vm-testqube-private' 20240522-122552
Saving to tlvm pool '/dev/qubes_dom0/vm-testqube-private'
Diff bytes: 28049408

Restoring VM settings:
 testqube
visible_netmask Def = 255.255.255.255
audiovm Def = dom0
start_time Def = 
kernel Def = 6.8.8-1.fc37
autostart Def = False
visible_ip Def = 10.137.0.11
gateway Def = 
updateable Def = False
gateway6 Def = 
template_for_dispvms Def = False
guivm Def = dom0
mac Def = 00:16:3e:5e:3c:00
vcpus = 8
maxmem = 4000
template = debian-12-xfce
keyboard_layout Def = us++
qrexec_timeout Def = 60
shutdown_timeout Def = 60
icon Def = appvm-blue
default_user Def = user
netvm Def = sys-firewall
provides_network Def = False
kernelopts Def = swiotlb=2048
visible_gateway6 Def = 
management_dispvm Def = default-mgmt-dvm
visible_gateway Def = 10.138.12.51
dns Def = 10.139.1.1 10.139.1.2
include_in_backups Def = True
debug Def = False
visible_ip6 Def = 
memory = 800
virt_mode Def = pvh
feature: menu-items = xfce4-file-manager.desktop firefox-esr.desktop libreoffice-calc.desktop libreoffice-impress.desktop libreoffice-startcenter.desktop libreoffice-writer.desktop org.gnome.Terminal.desktop
feature: qubesmanager.maxmem_value = 6000
['sudo', '-u', '#1000', 'qvm-appmenus', '--update', 'testqube']

[UndeadDevel]

To clarify: the keyboard_layout value does change when I change the keyboard layout and the qube is not affected; the issue is that the keyboard layout change does not propagate to affected qubes and restoring a qube from a wyng backup somehow makes them be affected / have this bug.

Edit: for affected qubes, I can manually change the keyboard layout on them using qvm-prefs, but that is, of course, much more cumbersome than it working via the keyboard shortcut that is supposed to propagate the new setting to all qubes. I'm not seeing anything in dom0 journalctl, either. I've also tried setting the keyboard layout to "per window" instead of "globally", but layout propagation still doesn't work for the qube when its window is in focus.

Edit2: restarting the qube or rebooting the system doesn't fix this, either, unfortunately.

Edit3: also tried removing the widget and setting Layout to system defaults; logging out and back in again after that...no change. Somehow it seems the qubes mechanism that iterates over qubes to change their layout when I switch it misses qubes restored by wyng...

[tasket]

@UndeadDevel I'm glad you posted the debug output. The util is setting keyboard_layout to its system default. That would mean there was no particular layout setting for your VM in the 'qubes.xml' at backup time. It does sound like you are using the desktop UI in dom0 to change layouts.

On my test VM, I set the keyboard_layout pref using qvm-prefs before backup, and the layout is restored properly. It is not getting set to the system default. Typing in the VM responds correctly to the layout I set.

I can try to change the layout from Qube Manager, but it says the function is unsupported.

And there is very little guidance on using keyboard layout in Qubes. @marmarek @DemiMarie can you comment on the disparity on UndeadDevel's machine?

[tasket]

@UndeadDevel Please try this:

Restore your test vm with wyng-util-qubes, then qvm-prefs -D vmname keyboard_layout. Does that allow the VM to respond to the layout widget?

[UndeadDevel]

Restore your test vm with wyng-util-qubes, then qvm-prefs -D vmname keyboard_layout. Does that allow the VM to respond to the layout widget?

That fixes it, thank you! I was hoping there was a way to fix it without having to move all my data to new qubes.

[tasket]

@UndeadDevel Good! Then the update in the '09beta' branch should avoid the problem without having to run qvm-prefs.