search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
529 stars 46 forks source link

upgrade from 4.0 to 4.1 breaks dom0 update -- [ERROR ] Failed to import module localemod #7114

Closed 1ofseven closed 2 years ago

1ofseven commented 2 years ago

Qubes OS release

Qubes R4.1 xen_version : 4.14.3 Linux 5.15.5-1.fc32.qubes.x86_64 Installed Packages:
grub2-qubes-theme.x86_64 5.14.4-2.fc32 kernel-latest-qubes-vm.x86_64 1000:5.14.15-1.fc25.qubes kernel-latest-qubes-vm.x86_64 1000:5.15.5-1.fc25.qubes kernel-latest-qubes-vm.x86_64 1000:5.15.5-1.fc32.qubes kernel-qubes-vm.x86_64 1000:5.4.151-1.fc25.qubes kernel-qubes-vm.x86_64 1000:5.4.156-1.fc25.qubes kernel-qubes-vm.x86_64 1000:5.10.76-1.fc32.qubes python2-qubesadmin.noarch 4.0.32-1.fc25 python2-qubesimgconverter.x86_64 4.0.31-1.fc25 python3-qubesadmin.noarch 4.1.18-1.fc32 python3-qubesdb.x86_64 4.1.11-1.fc32 python3-qubesimgconverter.x86_64 4.1.16-1.fc32 qubes-anaconda-addon.noarch 4.1.7-1.fc32 qubes-artwork.noarch 4.1.12-1.fc32 qubes-artwork-plymouth.noarch 4.1.12-1.fc32 qubes-audio-daemon.x86_64 4.1.19-1.fc32 qubes-audio-dom0.x86_64 4.1.19-1.fc32 qubes-core-admin-addon-whonix.noarch 4.0.2-1.fc32 qubes-core-admin-client.noarch 4.1.18-1.fc32 qubes-core-dom0.noarch 4.1.23-1.fc32 qubes-core-dom0-linux.x86_64 4.1.16-1.fc32 qubes-core-dom0-linux-kernel-install.x86_64 4.1.16-1.fc32 qubes-core-qrexec.x86_64 4.1.15-1.fc32 qubes-core-qrexec-dom0.x86_64 4.1.15-1.fc32 qubes-core-qrexec-libs.x86_64 4.1.15-1.fc32 qubes-db.x86_64 4.1.11-1.fc32 qubes-db-dom0.x86_64 4.1.11-1.fc32 qubes-db-libs.x86_64 4.1.11-1.fc32 qubes-desktop-linux-common.noarch 4.1.11-1.fc32 qubes-desktop-linux-manager.noarch 4.1.11-1.fc32 qubes-dist-upgrade.noarch 4.0.3-1.fc25 qubes-gpg-split-dom0.x86_64 2.0.54-1.fc32 qubes-gui-daemon.x86_64 4.1.19-1.fc32 qubes-gui-dom0.x86_64 4.1.19-1.fc32 qubes-img-converter-dom0.x86_64 1.2.11-1.fc32 qubes-input-proxy.x86_64 1.0.25-1.fc32 qubes-input-proxy-receiver.x86_64 1.0.25-1.fc32 qubes-libvchan-xen.x86_64 4.1.7-1.fc32 qubes-manager.noarch 4.1.22-1.fc32 qubes-menus.noarch 4.1.11-1.fc32 qubes-mgmt-salt.noarch 4.1.13-1.fc32 qubes-mgmt-salt-admin-tools.noarch 4.1.13-1.fc32 qubes-mgmt-salt-base.noarch 4.1.4-1.fc32 qubes-mgmt-salt-base-config.noarch 4.1.1-1.fc32 qubes-mgmt-salt-base-overrides-libs.noarch 4.0.2-1.fc25 qubes-mgmt-salt-base-topd.noarch 4.1.3-1.fc32 qubes-mgmt-salt-config.noarch 4.1.13-1.fc32 qubes-mgmt-salt-dom0.noarch 4.1.13-1.fc32 qubes-mgmt-salt-dom0-qvm.noarch 4.1.4-1.fc32 qubes-mgmt-salt-dom0-update.noarch 4.1.8-1.fc32 qubes-mgmt-salt-dom0-virtual-machines.noarch 4.1.15-1.fc32 qubes-pdf-converter-dom0.x86_64 2.1.12-1.fc32 qubes-release.noarch 4.1-0.27 qubes-release-notes.noarch 4.1-0.27 qubes-repo-templates.noarch 4.1.2-1.fc32 qubes-rpm-oxide.x86_64 0.2.2-1.fc32 qubes-usb-proxy-dom0.noarch 1.1.1-1.fc32 qubes-utils.x86_64 4.1.16-1.fc32 qubes-utils-libs.x86_64 4.1.16-1.fc32 xfce4-settings-qubes.x86_64 4.0.5-2.fc32

Brief summary

After upgrade from Qubes 4.0 to 4.1 an update of dom0 and templates returned the following warning and error:

[WARNING ] /usr/lib/python3.8/site-packages/salt/utils/files.py:396: RuntimeWarning: 
line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used

[ERROR ] Failed to import module localemod, this is due most likely to a syntax error

Steps to reproduce

Perform a version upgrade of Qubes from version 4.0 to 4.1 then either run Qubes Update from the menu or: sudo qubesctl --show-output state.sls update.qubes-dom0 sudo qubesctl --show-output --skip-dom0 --targets=debian-10,whonix-gw-16,whonix-ws-16 state.sls update.qubes-vm sudo qubesctl --show-output --skip-dom0 --targets=fedora-34,fedora-34-unt state.sls update.qubes-vm

Expected behavior

Updates complete without error.

Actual behavior

After upgrade from Qubes 4.0 to 4.1 an update of dom0 and templates returned the following warning and error:

[WARNING ] /usr/lib/python3.8/site-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used f_handle = open(*args, **kwargs) # pylint: disable=resource-leakage

[ERROR ] Failed to import module localemod, this is due most likely to a syntax error:
Traceback (most recent call last):
File “/usr/lib/python3.8/site-packages/salt/loader.py”, line 1685, in _load_module
mod = spec.loader.load_module()
File “”, line 522, in _check_name_wrapper
File “”, line 1027, in load_module
File “”, line 852, in load_module
File “”, line 265, in _load_module_shim
File “”, line 702, in _load
File “”, line 671, in _load_unlocked
File “”, line 848, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/var/cache/salt/minion/extmods/modules/localemod.py”, line 222, in
@decorators.which(‘locale-gen’)
AttributeError: module ‘salt.utils.decorators’ has no attribute ‘which’
1ofseven commented 2 years ago

Is this related to Whonix-16 support (or lack of) of Qubes? Not sure if it's related but, also getting an error during shutdown of a whonix disposable vm: "Denied: whonix.SdwdateStatus Denied whonix.SdwdateStatus from sys-whonix to disp####"

=integer assigned to dvm.

This occurs whether closing a browser or terminal. If this is doubling up on bugs, this can be made into a separate bug report.

1ofseven commented 2 years ago

Upgrade performed using: sudo qubes-dom0-update -y qubes-dist-upgrade

Not sure if this is applicable, but errors occurred at the end of STAGE 3:

"Error: Transaction check error: file /usr/share/bash-completion/completions/rfkill from install of util-linux-2.35.2-1.fc32.x86_64 conflicts with file from package bash-completion-1:2.5-1.fc25.noarch file /usr/bin/rst2html from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2html5 from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2latex from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2man from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2odt from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2odt_prepstyles from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2pseudoxml from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2s5 from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2xetex from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rst2xml from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /usr/bin/rstpep2html from install of python3-docutils-0.15.2-4.fc32.noarch conflicts with file from package python2-docutils-0.13.1-3.fc25.noarch file /etc/sysconfig/network-scripts/ifdown-ppp from install of network-scripts-ppp-2.4.7-35.fc32.x86_64 conflicts with file from package ppp-2.4.7-9.fc24.x86_64 file /etc/sysconfig/network-scripts/ifup-ppp from install of network-scripts-ppp-2.4.7-35.fc32.x86_64 conflicts with file from package ppp-2.4.7-9.fc24.x86_64 file /etc/sysconfig/network-scripts/ifdown-Team from install of network-scripts-teamd-1.31-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64 file /etc/sysconfig/network-scripts/ifdown-TeamPort from install of network-scripts-teamd-1.31-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64 file /etc/sysconfig/network-scripts/ifup-Team from install of network-scripts-teamd-1.31-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64 file /etc/sysconfig/network-scripts/ifup-TeamPort from install of network-scripts-teamd-1.31-2.fc32.x86_64 conflicts with file from package teamd-1.27-1.fc25.x86_64 file /usr/lib64/qt4/plugins/designer/libpyqt4.so from install of python3-PyQt4-4.12.3-11.fc32.x86_64 conflicts with file from package PyQt4-4.11.4-15.fc25.x86_64

Error Summary"

tlaurion commented 2 years ago

@marmarek @fepitre, also having OP described issue after running qubes-dist-upgrade just now.

I have disk clones of OEM disk image prior and after. Nobody else reported this after successful upgrade to 4.1?

marmarek commented 2 years ago

Haven't seen this one elsewhere. Maybe it's related to extra packages installed in dom0 before (although, it's unlikely, given you have util-linux on the list - which is installed always)? There was similar issue with python2-systemd, for which we have special handling in the script. What your updatevm is based on?

tlaurion commented 2 years ago

@marmarek updatevm based on fedora-34

marmarek commented 2 years ago

Hmm, that should work just fine...

Can you dump full list of packages installed in dom0 before the update (rpm -qa)? And the exact version of the qubes-dist-upgrade package used?

lyuts commented 2 years ago

Observing same issue after upgrade from 4.0 to 4.1. Update VM is sys-firewall DVM based on fedora 34. Template has been set up to comply with requirements of update VM from here (https://www.qubes-os.org/doc/templates/minimal/#customization).

Side note: trying to figure out what could have gone wrong I noticed that in Global settings -> Default kernel used by qubes was pointing to 5.4 from fc25. Changed that 5.10.96 from fc32.

Attempt to update dom0 spawns an xterm in the update VM, which tries to update repos, then prints a list of packages and terminates right away. I tried to capture this with the following command:

$ sudo qubes-dom0-update --clean --console --show-output
Using sys-firewall-dvm-34 as UpdateVM to download updates for Dom0; this may take some time...
warning: Converting database from bdb_ro to sqlite backend
0 files removed
Fedora 32 - x86_64                              4.2 MB/s |  70 MB     00:16    
Fedora 32 - x86_64 - Updates                    1.3 MB/s |  30 MB     00:23    
Qubes Dom0 Repository (updates)                 502 kB/s | 2.7 MB     00:05    
determining the fastest mirror (15 hosts).. done.--  B/s |   0  B     --:-- ETA
Qubes Templates repository                      1.5 kB/s | 4.3 kB     00:02    
Qubes Templates repository                      5.0 kB/s | 5.0 kB     00:01    
Dependencies resolved.
================================================================================
 Package                     Arch        Version             Repository    Size
================================================================================
Removing dependent packages:
 python2-backports_abc       noarch      0.5-1.fc25          @System       30 k
 python2-futures             noarch      3.1.1-1.fc25        @System       91 k
 python2-lxml                x86_64      4.1.1-1.fc25        @System      4.5 M
 python2-msgpack             x86_64      0.4.8-1.fc25        @System      276 k
 python2-nose                noarch      1.3.7-11.fc25       @System      1.1 M
 python2-pycurl              x86_64      7.43.0-6.fc25       @System      891 k
 python2-pygpgme             x86_64      0.3-18.fc25         @System      306 k
 python2-pysocks             noarch      1.6.7-1.fc25        @System       79 k
 python2-requests            noarch      2.10.0-4.fc25       @System      369 k
 python2-singledispatch      noarch      3.4.0.3-5.fc25      @System       46 k
 python2-systemd             x86_64      234-5.1.fc25        @System      263 k
 python2-tornado             x86_64      4.4.2-1.fc25        @System      3.9 M
 python2-urllib3             noarch      1.15.1-3.fc25       @System      443 k
 python2-zmq                 x86_64      15.3.0-2.fc25       @System      1.5 M

Transaction Summary
================================================================================
Remove  14 Packages

Freed space: 14 M
DNF will only download packages for the transaction.
Complete!
No packages downloaded
No updates available
lyuts commented 2 years ago

Hmm, that should work just fine...

Can you dump full list of packages installed in dom0 before the update (rpm -qa)? And the exact version of the qubes-dist-upgrade package used?

I'll let others share their dom0 packages list too, but here's mine in the attachment. dom0.pkg.list.tar.gz

tlaurion commented 2 years ago

Exactly the same output from 4.1. Output of 4.0 packages will follow in next days @marmarek

lyuts commented 2 years ago

Out of hypothesis that my update vm ran out of disk space I tried to update individual package with sudo qubes-dom0-update --clean --console --show-output python2-zmq and in addition to the output i provided above i get this error:

Qubes OS Repository for Dom0                                 0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'qubes-dom0-cached':
  - Curl error (37): Couldn't read a file:// file for file:///var/lib/qubes/updates/repodata/repomd.xml [Couldn't open file /var/lib/qubes/updates/repodata/repomd.xml]
Error: Failed to download metadata for repo 'qubes-dom0-cached': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
mfc commented 2 years ago

did in-place upgrade 4.0->4.1 to avoid the unclear Heads upgrade flashing instructions and encountered same issue as OP.

i don't know if it's related but booting with default kernel freezes/times-out (if interested, lemme know how to get logs to share):

and i need to select the 2nd boot option:

however once in Qubes i can run individual qubes with 5.10.96-1.fc32 kernel fine.

(fixed)

tlaurion commented 2 years ago

Hmmm I will try to make more time into troubleshooting this and #7122 which are blockers to actually push users into upgrading before Whonix EOL of 03/22 on Qubes 4.0.4

@adrelanos @marmarek: Delays are really short for users. Really surprised by 4.1 release having came so fast. Now we expect users to reinstall and restore backup on top of 4.1 it seems.

tlaurion commented 2 years ago

@mfc you figured how to resolve OP issue?

mfc commented 2 years ago

i get same issues as here: https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1038379418

not sure how significant these issues are, Qubes is otherwise usable.

tlaurion commented 2 years ago

Hmm, that should work just fine...

@marmarek : will edit this post as I go. Can you dump full list of packages installed in dom0 before the update (rpm -qa)?

List of packages installed prior of qubes-dist-upgrade: Q40_rpm_qa_sorted.txt

And the exact version of the qubes-dist-upgrade package used?

qubes-dist-upgrade-4.0.4-1.fc25.noarch

Steps: 1- Cloning disk at that point to be able to redo as needed to troubleshoot exactly that state.... (This is where I would love to have wyng<->qubes remote backups to share with you here, to ease troubleshooting and ease revertible, shared states of dom0 and everything else.... I would only have to say: apply that XYZ tagged state to a testing machine: Q40_Q41_upgrade_test_March012022. But... no). 2- All qubes-dist-upgrade steps (console logs) here: qubes-dist-upgrade-all.txt

tlaurion commented 2 years ago

Redoing the upgrade from 4.0 today doesn't show that behavior anymore....

lyuts commented 2 years ago

Question for knowledgeable folks, are python packages listed https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1037523549 safe to remove? Is it correct that R4.1 moved to using to python3 and python2 variants are no longer needed?

DemiMarie commented 2 years ago

Question for knowledgeable folks, are python packages listed #7114 (comment) safe to remove? Is it correct that R4.1 moved to using to python3 and python2 variants are no longer needed?

Correct, my system has no python2 packages installed.

mfc commented 2 years ago

i get same issues as here: #7114 (comment)

not sure how significant these issues are, Qubes is otherwise usable.

i didn't realize but this issue isn't just related to the upgrade process but also makes dom0 updates fail (with same error as OP):

Updating dom0

[ERROR   ] Failed to import module localemod, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/salt/loader.py", line 1685, in _load_module
    mod = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 522, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1027, in load_module
  File "<frozen importlib._bootstrap_external>", line 852, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/cache/salt/minion/extmods/modules/localemod.py", line 222, in <module>
    @decorators.which('locale-gen')
AttributeError: module 'salt.utils.decorators' has no attribute 'which'
local:
    ----------

no issue with updating templates.

mfc commented 2 years ago

and just to add, even tho dom0 reports an ERROR, the Qubes Updater gives the user a nice green check. only by looking at Details does the user find it actually failed.

other user reports: https://forum.qubes-os.org/t/qubes-users-dom0-update-broken-after-upgrade-to-4-1/9238

i have renamed the issue to communicate more clearly why this is an issue.

marmarek commented 2 years ago

localemod.py shouldn't be installed in 4.1 anymore. That's leftover qubes-mgmt-salt-base-overrides package that wasn't removed during upgrade.

marmarek commented 2 years ago

@mfc can you confirm whether removing qubes-mgmt-salt-base-overrides fixes it?

mfc commented 2 years ago

not finding the package in dom0:

$ sudo dnf remove qubes-mgmt-salt-base-overrides
No match for argument: qubes-mgmt-salt-base-overrides
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
marmarek commented 2 years ago

Maybe it was removed then, but some cached copy remained. Try:

qubesctl saltutil.clear_cache
qubesctl saltutil.sync_all
mfc commented 2 years ago 
$ sudo qubesctl saltutil.clear_cache
local:
    True
$ sudo qubesctl saltutil.sync_all
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
local:
    ----------
    beacons:
    clouds:
    engines:
    executors:
    grains:
        - grains.boot_mode
        - grains.pci_devs
        - grains.redefined_dom0_grains
        - grains.whonix
    log_handlers:
    matchers:
    modules:
        - modules.debug
        - modules.ext_module_qvm
        - modules.module_utils
        - modules.qubes
        - modules.qubes_dom0_update
        - modules.topd
    output:
    pillar:
        - pillar.qvm_prefs
    proxymodules:
    renderers:
    returners:
    sdb:
    serializers:
    states:
        - states.debug
        - states.ext_state_qvm
        - states.status
    thorium:
    utils:
        - utils.__init__
        - utils.fileinfo
        - utils.matcher
        - utils.nulltype
        - utils.pathinfo
        - utils.pathutils
        - utils.qubes_utils
        - utils.toputils
$ sudo qubes-dom0-update --clean --console --show-output
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
0 files removed
Unable to detect release version (use '--releasever' to specify release version)
Fedora 32 - x86_64 - Updates                    2.4 MB/s |  35 MB     00:14    
Fedora 32 - x86_64                              2.7 MB/s |  63 MB     00:23    
Qubes Dom0 Repository (updates)                 657 kB/s | 2.9 MB     00:04    
determining the fastest mirror (15 hosts).. done.--  B/s |   0  B     --:-- ETA
Qubes Templates repository                      2.2 kB/s | 4.3 kB     00:01    
Last metadata expiration check: 0:00:01 ago on Sat Mar 12 11:36:14 2022.
Dependencies resolved.
================================================================================
 Package                     Arch        Version             Repository    Size
================================================================================
Removing dependent packages:
 python2-backports_abc       noarch      0.5-1.fc25          @System       30 k
 python2-futures             noarch      3.1.1-1.fc25        @System       91 k
 python2-lxml                x86_64      4.1.1-1.fc25        @System      4.5 M
 python2-msgpack             x86_64      0.4.8-1.fc25        @System      276 k
 python2-nose                noarch      1.3.7-11.fc25       @System      1.1 M
 python2-pycurl              x86_64      7.43.0-6.fc25       @System      891 k
 python2-pygpgme             x86_64      0.3-18.fc25         @System      306 k
 python2-pysocks             noarch      1.6.7-1.fc25        @System       79 k
 python2-requests            noarch      2.10.0-4.fc25       @System      369 k
 python2-singledispatch      noarch      3.4.0.3-5.fc25      @System       46 k
 python2-systemd             x86_64      234-5.1.fc25        @System      263 k
 python2-tornado             x86_64      4.4.2-1.fc25        @System      3.9 M
 python2-urllib3             noarch      1.15.1-3.fc25       @System      443 k
 python2-zmq                 x86_64      15.3.0-2.fc25       @System      1.5 M

Transaction Summary
================================================================================
Remove  14 Packages

Freed space: 14 M
DNF will only download packages for the transaction.
Complete!
No packages downloaded
No updates available
$ sudo qubesctl --show-output state.sls update.qubes-dom0
[WARNING ] /usr/lib/python3.8/site-packages/salt/utils/files.py:396: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage

local:
----------
          ID: /etc/yum.repos.d/qubes-dom0.repo
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 11:37:13.004464
    Duration: 16.295 ms
     Changes:   
----------
          ID: /etc/yum.repos.d/qubes-templates.repo
    Function: file.replace
      Result: True
     Comment: No changes needed to be made
     Started: 11:37:13.020927
    Duration: 4.492 ms
     Changes:   
----------
          ID: update
    Function: cmd.run
        Name: if qubes-dom0-update --quiet --assumeyes --clean --action=clean expire-cache >/dev/null 2>&1; then
    echo "result=True comment='Cache cleaned'"
else
    echo "result=False comment='Could not clean cache'"
fi

      Result: True
     Comment: Cache cleaned
     Started: 11:37:13.026787
    Duration: 14310.702 ms
     Changes:   
----------
          ID: update
    Function: pkg.uptodate
      Result: True
     Comment: System is already up-to-date
     Started: 11:37:29.382735
    Duration: 159241.032 ms
     Changes:   

Summary for local
------------
Succeeded: 4
Failed:    0
------------
Total states run:     4
Total run time: 173.573 s
mfc commented 2 years ago 
$ sudo dnf remove python2
Dependencies resolved.
=====================================================================================================================
 Package                                     Architecture   Version                 Repository                  Size
=====================================================================================================================
Removing:
 python27                                    x86_64         2.7.18-8.fc32           @qubes-dom0-cached          55 M
Removing dependent packages:
 PyYAML                                      x86_64         3.11-13.fc25            @anaconda                  629 k
 dbus-python                                 x86_64         1.2.4-2.fc25            @anaconda                  482 k
 pyliblzma                                   x86_64         0.5.3-16.fc25           @anaconda                  192 k
 python-xpyb                                 x86_64         1.3.1-6.fc24            @anaconda                  1.2 M
 python2-docutils                            noarch         0.15.2-4.fc32           @qubes-dom0-cached         6.3 M
 python2-futures                             noarch         3.1.1-1.fc25            @anaconda                   91 k
 python2-jinja2                              noarch         2.11.3-1.fc32           @qubes-dom0-cached         2.6 M
 python2-lxml                                x86_64         4.1.1-1.fc25            @anaconda                  4.5 M
 python2-msgpack                             x86_64         0.4.8-1.fc25            @anaconda                  276 k
 python2-nose                                noarch         1.3.7-11.fc25           @anaconda                  1.1 M
 python2-psutil                              x86_64         5.6.7-1.fc32            @qubes-dom0-cached         2.3 M
 python2-pycurl                              x86_64         7.43.0-6.fc25           @anaconda                  891 k
 python2-pygpgme                             x86_64         0.3-18.fc25             @anaconda                  306 k
 python2-pysocks                             noarch         1.6.7-1.fc25            @anaconda                   79 k
 python2-qubesadmin                          noarch         4.0.32-1.fc25           @qubes-dom0-cached         618 k
 python2-qubesimgconverter                   x86_64         4.0.31-1.fc25           @qubes-dom0-cached          40 k
 python2-requests                            noarch         2.10.0-4.fc25           @anaconda                  369 k
 python2-systemd                             x86_64         234-5.1.fc25            @qubes-dom0-cached         263 k
 python2-tornado                             x86_64         4.4.2-1.fc25            @anaconda                  3.9 M
 python2-zmq                                 x86_64         15.3.0-2.fc25           @anaconda                  1.5 M
 pyxattr                                     x86_64         0.5.3-8.fc25            @anaconda                   63 k
 qubes-mgmt-salt-base-overrides-libs         noarch         4.0.2-1.fc25            @anaconda                   20 k
 yum-metadata-parser                         x86_64         1.1.4-17.fc25           @anaconda                   74 k
Removing unused dependencies:
 python2-babel                               noarch         2.8.0-4.fc32            @qubes-dom0-cached          26 M
 python2-backports_abc                       noarch         0.5-1.fc25              @anaconda                   30 k
 python2-cairo                               x86_64         1.18.2-4.fc32           @qubes-dom0-cached         323 k
 python2-markupsafe                          x86_64         1.1.1-5.fc32            @qubes-dom0-cached          65 k
 python2-numpy                               x86_64         1:1.16.4-7.fc32         @qubes-dom0-cached          19 M
 python2-olefile                             noarch         0.46-9.fc32             @qubes-dom0-cached         256 k
 python2-pillow                              x86_64         6.2.2-5.fc32            @qubes-dom0-cached         2.4 M
 python2-setuptools                          noarch         41.2.0-2.fc32           @qubes-dom0-cached         3.1 M
 python2-singledispatch                      noarch         3.4.0.3-5.fc25          @anaconda                   46 k
 python2-urllib3                             noarch         1.15.1-3.fc25           @anaconda                  443 k
 python3-wrapt                               x86_64         1.11.2-5.fc32           @qubes-dom0-cached         166 k
 pytz                                        noarch         2016.6.1-1.fc25         @anaconda                  186 k
 tix                                         x86_64         1:8.4.3-27.fc31         @qubes-dom0-cached         1.0 M

Transaction Summary
=====================================================================================================================
Remove  37 Packages

Freed space: 136 M
Is this ok [y/N]:

in proposing removing python2 an interesting-sounding package pops up, qubes-mgmt-salt-base-overrides-libs

marmarek commented 2 years ago

in proposing removing python2 an interesting-sounding package pops up, qubes-mgmt-salt-base-overrides-libs

That is correct, it shouldn't be needed anymore (but also shouldn't hurt).

mfc commented 2 years ago

is this a relevant error?

$ sudo qubesctl saltutil.sync_all
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable

i removed qubes-mgmt-salt-base-overrides-libs, also removed all python2 packages that are listed from sudo qubes-dom0-update. i haven't deleted all the python2-suggested packages since some sound still relevant (yum-metadata-parser, dbus-python).

running sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing works, so dom0 updates work :tada: reboots fine.

marmarek commented 2 years ago

[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable

That's effect of clear_cache, the first call after that does not have "qvm_prefs" module loaded yet. Nothing to worry about.

tlaurion commented 2 years ago

I see manual interventions needed from end users, while no more user report for this failure (and was not able to replicate https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1048943587 7 days later https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1055850571.

Not sure why the bug just vanished, but some applied fixes resolved the issue since then @marmarek ?

marmarek commented 2 years ago

Updating some salt-related packages implicitly clears the cache.

mfc commented 2 years ago

i may have also been wrong thinking that dom0 updates were blocked by this

tlaurion commented 2 years ago

@mfc @marmarek only testing would say.

DistractedSorbet commented 2 years ago

I came here to say this is still an issue. I upgraded from 4.0 to 4.1 today, my first attempt at doing so. I ran into some issues along the way and had to restart the process a few times, but ultimately, the process completed and I restarted.

My current situation is that Dom0, template, and standalone updates all fail with various errors. In other words, I can't update anything post-upgrade to 4.1. Dom0 updates, in particular, fail with the exact same error as reported in the original bug and by @mfc, down to the line numbers (https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1062762215).

I have also encountered this erroneous green checkmark output mentioned above as well: https://github.com/QubesOS/qubes-issues/issues/7114#issuecomment-1065098164.

I have no idea if the errors encountered when attempting to update templates or standalones are a result of the Dom0 issue, or separate unrelated errors. I'd like to fix Dom0 first and see what's still broken after that.

I came to Qubes OS in version 4.0, so the only way I know to update safely is using the Qubes Update tool. I don't trust my understanding enough to start running some of the commands I see in this thread in Dom0 to "see if it fixes the problem". Can someone clearly describe the workaround procedure (if one exists) in a step-by-step fashion?

Osndok commented 2 years ago

Are we sure that this message is coming from dom0? Maybe it is a problem on the UpdateVM side?

RhizomaticNomad commented 2 years ago

As DistractedSorbet I also felt into this problem while upgrading from 4.0 to 4.1. Happily I was able to resolve this with the suggestions in this thread, which was the correcting setting in the end I can't say. Following actions I performed:

in Global settings -> Default kernel used by qubes was pointing to 5.4 from fc25, changed that to 5.10.109-1.fc32.

sudo dnf remove qubes-mgmt-salt-base-overrides (not necessary, because it wasn't installed (anymore?))

qubesctl saltutil.clear_cache
qubesctl saltutil.sync_all

After this, the Dom0 could be updated again. Except whonix gw and ws, where there is still the error about missing attribute 'os_family' in the Jinja template, but that's another topic ...

qubesos-bot commented 2 years ago

Automated announcement from builder-github

The component dist-upgrade (including package qubes-dist-upgrade-4.0.6-1.fc25) has been pushed to the r4.0 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

akoskoronka commented 2 years ago

I have the same dom0-update problem after upgrading 4.0 to 4.1 with qubes-dist-upgrade-4.0.4-1.fc25. As everyone did, I also removed python2, python2-systemd, qubes-mgmt-salt-base-overrides, changed default kernel in Global settings, did the clear_cache and sync_all, but qubes-dom0-update still gives error during transaction test: Transaction check succeeded. Running transaction test Error: Transaction test error: file /usr/lib.build-id/10/426e215ebe....0a68 from install of kernel-devel-1000:5.10.109-1.fc32.qubes.x86_64 conflicts with file from package kernel-devel-1000:5.10.96-1.fc32.qubes.x86_64 ...

Tried running qubes-dist-upgrade again to see if there is anything it didn't finish on previous run, now it fails at step3:

Error: Unable to find a match: python?-systemd -> Launch restoration... ... done

Please note, that python3-systemd is still installed.

After this I've dnf removed qubes-dist-upgrade, and cannot reinstall (package is only available in 4.0 repo) due to the partially successful dist upgrade. I could reinstall this pkg manually and play with it if it helps debugging this problem, otherwise I will simply make a fresh 4.1 install.

Thanks!

qubesos-bot commented 2 years ago

Automated announcement from builder-github

The component dist-upgrade (including package qubes-dist-upgrade-4.0.6-1.fc25) has been pushed to the r4.0 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

akoskoronka commented 2 years ago

After removing kernel-devel, the dom0 update was successful.

qtpies commented 2 years ago

in proposing removing python2 an interesting-sounding package pops up, qubes-mgmt-salt-base-overrides-libs

That is correct, it shouldn't be needed anymore (but also shouldn't hurt).

Used this to solve issue #7503 (see comment there for details+package removal log). Immediately after I can't upgrade both debian-11 and fedora-35 templates anymore with qubes-update. Errors are identical for both templates. Any suggestion how to continue?

First:

[user@dom0 ~]$ sudo qubesctl --skip-dom0 --targets=debian-11 --show-output state.sls update.qubes-vm
debian-11:
  /etc/qubes-rpc/qubes.SaltLinuxVM: line 46: salt-ssh: command not found

Tried to fix this by:

user@debian-11:~$ sudo apt install salt-ssh
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  python3-croniter python3-natsort python3-psutil python3-tz python3-tzlocal python3-zmq salt-common
Suggested packages:
  python-natsort-doc python-psutil-doc python3-mako salt-doc
The following NEW packages will be installed:
  python3-croniter python3-natsort python3-psutil python3-tz python3-tzlocal python3-zmq salt-common salt-ssh

Result:

[user@dom0 ~]$ sudo qubesctl --skip-dom0 --targets=debian-11 --show-output state.sls update.qubes-vm
debian-11:
      ssh: Could not resolve hostname debian-11: Temporary failure in name resolution
tzwcfq commented 2 years ago

Maybe you're missing qubes-mgmt-salt-vm-connector package. If not then check if you're missing some of the qubes packages. I have these packages in debian-11 template:

Expand: ``` $ sudo dpkg -l | grep qubes ii libqubes-rpc-filecopy2 4.1.16+deb11u1 amd64 Qubes file copy protocol library ii libqubesdb 4.1.13-1+deb11u1 amd64 QubesDB libs. ii pulseaudio-qubes 4.1.25-1+deb11u1 amd64 Audio support for Qubes VM ii python3-qubesdb 4.1.13-1+deb11u1 amd64 QubesDB python bindings. ii python3-qubesimgconverter 4.1.16+deb11u1 amd64 Python package qubesimgconverter ii qubes-core-agent 4.1.35-1+deb11u1 amd64 Qubes core agent ii qubes-core-agent-dom0-updates 4.1.35-1+deb11u1 amd64 Scripts required to handle dom0 updates. ii qubes-core-agent-nautilus 4.1.35-1+deb11u1 amd64 Qubes integration for Nautilus ii qubes-core-agent-network-manager 4.1.35-1+deb11u1 amd64 NetworkManager integration for Qubes VM ii qubes-core-agent-networking 4.1.35-1+deb11u1 amd64 Networking support for Qubes VM ii qubes-core-agent-passwordless-root 4.1.35-1+deb11u1 amd64 Passwordless root access from normal user ii qubes-core-qrexec 4.1.18-1+deb11u1 amd64 Qubes qrexec agent ii qubes-gpg-split 2.0.58-1+deb11u1 amd64 The Qubes service for secure gpg separation ii qubes-gui-agent 4.1.25-1+deb11u1 amd64 Makes X11 windows available to qubes dom0 ii qubes-img-converter 1.2.11-1+deb11u1 amd64 Qubes service for converting untrusted images into trusted ones. ii qubes-input-proxy-sender 1.0.26-1+deb11u1 amd64 Provides Simple input events proxy ii qubes-kernel-vm-support 4.1.16+deb11u1 amd64 Qubes VM kernel and initramfs modules ii qubes-mgmt-salt-vm-connector 4.1.13-1+deb11u1 all Interface for managing VM from dom0 ii qubes-pdf-converter 2.1.12-1+deb11u1 amd64 The Qubes service for converting untrusted PDF files into trusted ones ii qubes-usb-proxy 1.1.1+deb11u1 amd64 USBIP wrapper to run it over Qubes RPC connection ii qubes-utils 4.1.16+deb11u1 amd64 Qubes Linux utilities ii qubes-vm-dependencies 4.1.20-1+deb11u1 amd64 Meta package with packages required in Qubes VM ii qubes-vm-recommended 4.1.20-1+deb11u1 amd64 Meta package with packages recommended in Qubes VM ii qubesdb 4.1.13-1+deb11u1 amd64 QubesDB management tools and daemon. ii qubesdb-vm 4.1.13-1+deb11u1 amd64 QubesDB VM service. ii xserver-xorg-input-qubes 4.1.25-1+deb11u1 amd64 X input driver for injecting events from qubes-gui-agent ii xserver-xorg-qubes-common 4.1.25-1+deb11u1 amd64 Common functions for qubes xserver driver ii xserver-xorg-video-dummyqbs 4.1.25-1+deb11u1 amd64 Dummy X video driver for qubes-gui-agent ```
qtpies commented 2 years ago

Thanks @tzwcfq. I wasn't missing that package but it was old. version 4.0.24, even though I do my updates. Dependency issue?Also some optional packages were not present (didn't save the old dpkg -l). I tried installing qubes-vm-recommended in the Debian template. This updates qubes-mgmt-salt-vm-connector + to 4.1.13 and updates/installs packages, log below.

After this, I can update both Debian and Fedora templates again, likely because Debian is my default-mgmt-dvm.

The old version of qubes-mgmt-salt-vm-connector seems the likely issue. It now requires salt-ssh but maybe it didn't before? Small sidenote: should salt-ssh in qubes-vm-recommended when it is already installed because it is a dependency of qubes-mgmt-salt-vm-connector, which is not an optional package? 

Commandline: apt install qubes-vm-recommended
Requested-By: user (1000)
Install: qubes-img-converter:amd64 (1.2.11-1+deb11u1, automatic), python3-colorama:amd64 (0.4.4-1, automatic), libxfce4util-bin:amd64 (4.16.0-1, automatic), python3-tz:amd64 (2021.1-1, automatic), libxfce4ui-common:amd64 (4.16.0-1, automatic), libxfce4ui-2-0:amd64 (4.16.0-1, automatic), libxfconf-0-3:amd64 (4.16.0-2, automatic), libxfce4panel-2.0-4:amd64 (4.16.2-1, automatic), python3-qubesimgconverter:amd64 (4.1.16+deb11u1, automatic), python3-psutil:amd64 (5.8.0-1, automatic), python3-click:amd64 (7.1.2-1, automatic), python3-numpy:amd64 (1:1.19.5-1, automatic), python3-croniter:amd64 (0.3.34-3, automatic), xfconf:amd64 (4.16.0-2, automatic), python3-natsort:amd64 (7.1.0-1, automatic), salt-common:amd64 (3002.6+dfsg1-4+deb11u1, automatic), python3-tzlocal:amd64 (2.1-1, automatic), python3-zmq:amd64 (20.0.0-1+b1, automatic), libxfce4util7:amd64 (4.16.0-1, automatic), qubes-mgmt-salt-vm-connector:amd64 (4.1.13-1+deb11u1, automatic), qubes-pdf-converter:amd64 (2.1.12-1+deb11u1, automatic), salt-ssh:amd64 (3002.6+dfsg1-4+deb11u1, automatic), qubes-vm-recommended:amd64 (4.1.20-1+deb11u1), python3-tqdm:amd64 (4.57.0-2, automatic), xfce4-notifyd:amd64 (0.6.2-1, automatic), libxfce4util-common:amd64 (4.16.0-1, automatic)
qubesos-bot commented 1 year ago

Automated announcement from builder-github

The package mgmt-salt-base has been pushed to the r4.1 testing repository for the CentOS centos-stream8 template. To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

qubesos-bot commented 1 year ago

Automated announcement from builder-github

The package qubes-mgmt-salt-base_4.1.5-1 has been pushed to the r4.1 testing repository for the Debian template. To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot commented 1 year ago

Automated announcement from builder-github

The package qubes-mgmt-salt-base_4.1.5-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template. To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot commented 1 year ago

Automated announcement from builder-github

The package mgmt-salt-base has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template. To install this update, please use the standard update command:

sudo yum update

Changes included in this update

qubesos-bot commented 1 year ago

Automated announcement from builder-github

The component mgmt-salt-base (including package qubes-mgmt-salt-base-4.1.5-1.fc32) has been pushed to the r4.1 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot commented 1 year ago

Automated announcement from builder-github

The component mgmt-salt-base (including package qubes-mgmt-salt-base-4.1.5-1.fc32) has been pushed to the r4.1 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update