search

ben-grande / qusal

Salt Formulas for Qubes OS.
14 stars 6 forks source link

sys-cacher.install-client fails in Fedora 40 Minimal #69

Closed c0mmando closed 1 week ago

c0mmando commented 1 week ago

Commitment

I confirm that I have read the following resources:

Software version

f5528fe

Brief summary

Installing sys-cacher client for Fedora qubes fails on a fresh install

Steps to reproduce

Fresh install and iterate through bootstrap steps. The following step will fail with the below log errors:

sudo qubesctl --skip-dom0 --targets="$(qvm-ls --no-spinner --raw-list --tags updatevm-sys-cacher | tr "\n" ",")" state.apply sys-cacher.install-client

errors:

2024-06-23 18:33:07,092 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/netutil.py", line 57, in <module>
2024-06-23 18:33:07,092 output:             import backports.ssl_match_hostname
2024-06-23 18:33:07,092 output:         ModuleNotFoundError: No module named 'backports'
2024-06-23 18:33:07,092 output:     stdout:
2024-06-23 18:33:07,092 exit code: 20

Expected behavior

Successful apt cacher client install and proceed with system bootstrap

Actual behavior

2024-06-23 18:32:38,551 calling 'state.apply sys-cacher.install-client'...
2024-06-23 18:33:07,088 output: fedora-40-minimal:
2024-06-23 18:33:07,088 output:     ----------
2024-06-23 18:33:07,088 output:     _error:
2024-06-23 18:33:07,088 output:         Failed to return clean data
2024-06-23 18:33:07,088 output:     retcode:
2024-06-23 18:33:07,088 output:         1
2024-06-23 18:33:07,088 output:     stderr:
2024-06-23 18:33:07,088 output:         Traceback (most recent call last):
2024-06-23 18:33:07,088 output:           File "/var/tmp/.root_dd8a91_salt/salt-call", line 27, in <module>
2024-06-23 18:33:07,089 output:             salt_call()
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/scripts.py", line 437, in salt_call
2024-06-23 18:33:07,089 output:             import salt.cli.call
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/call.py", line 3, in <module>
2024-06-23 18:33:07,089 output:             import salt.cli.caller
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/caller.py", line 12, in <module>
2024-06-23 18:33:07,089 output:             import salt.channel.client
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/channel/client.py", line 13, in <module>
2024-06-23 18:33:07,089 output:             import salt.crypt
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/crypt.py", line 26, in <module>
2024-06-23 18:33:07,089 output:             import salt.payload
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/payload.py", line 12, in <module>
2024-06-23 18:33:07,089 output:             import salt.loader.context
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/__init__.py", line 15, in <module>
2024-06-23 18:33:07,089 output:             import salt.config
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/config/__init__.py", line 109, in <module>
2024-06-23 18:33:07,089 output:             _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
2024-06-23 18:33:07,089 output:                                     ^^^^^^^^^^^^^^^^^^^^^^
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/config/__init__.py", line 97, in _gather_buffer_space
2024-06-23 18:33:07,089 output:             import salt.grains.core
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/grains/core.py", line 30, in <module>
2024-06-23 18:33:07,089 output:             import salt.modules.cmdmod
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
2024-06-23 18:33:07,089 output:             import salt.utils.templates
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/templates.py", line 21, in <module>
2024-06-23 18:33:07,089 output:             import salt.utils.http
2024-06-23 18:33:07,089 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/http.py", line 27, in <module>
2024-06-23 18:33:07,089 output:             import salt.ext.tornado.simple_httpclient
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/simple_httpclient.py", line 9, in <module>
2024-06-23 18:33:07,090 output:             from salt.ext.tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/http1connection.py", line 31, in <module>
2024-06-23 18:33:07,090 output:             from salt.ext.tornado import iostream
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/iostream.py", line 41, in <module>
2024-06-23 18:33:07,090 output:             from salt.ext.tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/netutil.py", line 57, in <module>
2024-06-23 18:33:07,090 output:             import backports.ssl_match_hostname
2024-06-23 18:33:07,090 output:         ModuleNotFoundError: No module named 'backports'
2024-06-23 18:33:07,090 output:         [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
2024-06-23 18:33:07,090 output:         ModuleNotFoundError: No module named 'backports'
2024-06-23 18:33:07,090 output:         Traceback (most recent call last):
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/salt-call", line 27, in <module>
2024-06-23 18:33:07,090 output:             salt_call()
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/scripts.py", line 437, in salt_call
2024-06-23 18:33:07,090 output:             import salt.cli.call
2024-06-23 18:33:07,090 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/call.py", line 3, in <module>
2024-06-23 18:33:07,091 output:             import salt.cli.caller
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/caller.py", line 12, in <module>
2024-06-23 18:33:07,091 output:             import salt.channel.client
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/channel/client.py", line 13, in <module>
2024-06-23 18:33:07,091 output:             import salt.crypt
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/crypt.py", line 26, in <module>
2024-06-23 18:33:07,091 output:             import salt.payload
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/payload.py", line 12, in <module>
2024-06-23 18:33:07,091 output:             import salt.loader.context
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/__init__.py", line 15, in <module>
2024-06-23 18:33:07,091 output:             import salt.config
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/config/__init__.py", line 109, in <module>
2024-06-23 18:33:07,091 output:             _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
2024-06-23 18:33:07,091 output:                                     ^^^^^^^^^^^^^^^^^^^^^^
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/config/__init__.py", line 97, in _gather_buffer_space
2024-06-23 18:33:07,091 output:             import salt.grains.core
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/grains/core.py", line 30, in <module>
2024-06-23 18:33:07,091 output:             import salt.modules.cmdmod
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
2024-06-23 18:33:07,091 output:             import salt.utils.templates
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/templates.py", line 21, in <module>
2024-06-23 18:33:07,091 output:             import salt.utils.http
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/http.py", line 27, in <module>
2024-06-23 18:33:07,091 output:             import salt.ext.tornado.simple_httpclient
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/simple_httpclient.py", line 9, in <module>
2024-06-23 18:33:07,091 output:             from salt.ext.tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/http1connection.py", line 31, in <module>
2024-06-23 18:33:07,091 output:             from salt.ext.tornado import iostream
2024-06-23 18:33:07,091 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/iostream.py", line 41, in <module>
2024-06-23 18:33:07,091 output:             from salt.ext.tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
2024-06-23 18:33:07,092 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/netutil.py", line 57, in <module>
2024-06-23 18:33:07,092 output:             import backports.ssl_match_hostname
2024-06-23 18:33:07,092 output:         ModuleNotFoundError: No module named 'backports'
2024-06-23 18:33:07,092 output:     stdout:
2024-06-23 18:33:07,092 exit code: 20
ben-grande commented 1 week ago

Please follow these steps: TROUBLESHOOT.md#get-salt-management-information.

c0mmando commented 1 week ago

Please follow these steps: TROUBLESHOOT.md#get-salt-management-information.

Global management_dispvm qube: default-mgmt-dvm
Global management_dispvm template: debian-12-xfce
Global management_dispvm template features:
template-name                                  debian-12-xfce
template-release                               202312171103
os-distribution                                debian
os-version                                     12
Wanted qube: fedora-40-minimal
Global management_dispvm qube: default-mgmt-dvm

this is the Qubes default management VM after a fresh install

ben-grande commented 1 week ago

Fedora Minimal can't be targeted by Debian management qube.

Switch the template of the management qube to the Full Fedora Template or run our "mgmt" formula, it will take care of producing a mgmt qube that works.

On Wed, Jun 26, 2024, 3:00 AM c0mmando @.***> wrote:

Please follow these steps: TROUBLESHOOT.md#get-salt-management-information https://github.com/ben-grande/qusal/blob/main/docs/TROUBLESHOOT.md#get-salt-management-information .

Global management_dispvm qube: default-mgmt-dvm Global management_dispvm template: debian-12-xfce Global management_dispvm template features: template-name debian-12-xfce template-release 202312171103 os-distribution debian os-version 12 Wanted qube: fedora-40-minimal Global management_dispvm qube: default-mgmt-dvm

this is the Qubes default management VM after a fresh install

— Reply to this email directly, view it on GitHub https://github.com/ben-grande/qusal/issues/69#issuecomment-2190316651, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCE2O4KIDPH5D2ZBQHK7Q2DZJIHCJAVCNFSM6AAAAABJYUIQTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJQGMYTMNRVGE . You are receiving this because you commented.Message ID: @.***>

c0mmando commented 1 week ago

Thanks for your response. I'll install mgmt before sys cacher from now on in the bootstrapping processes and see if that solves the issue. You may want to swap the install order in the bootstrap guide.

ben-grande commented 1 week ago

Pushing a commit to reorder the bootstrap guide. Please report back if everything went fine or not.

c0mmando commented 1 week ago

Everything went fine with Fedora, however debian-12-minimal threw some errors during the sys-cacher install script. debian-12-xfce worked without issue.

$ qvm-mgmt debian-12-minimal
Global management_dispvm qube: dvm-mgmt
Global management_dispvm template: tpl-mgmt
Global management_dispvm template features:
os-distribution                             fedora
os-version                                  40
template-name                               fedora-40-minimal
template-release                            202404260902
Wanted qube: debian-12-minimal
Global management_dispvm qube: dvm-mgmt

$ qvm-mgmt debian-12-xfce
Global management_dispvm qube: dvm-mgmt
Global management_dispvm template: tpl-mgmt
Global management_dispvm template features:
os-distribution                             fedora
os-version                                  40
template-name                               fedora-40-minimal
template-release                            202404260902
Wanted qube: debian-12-xfce
Global management_dispvm qube: dvm-mgmt

2024-06-27 02:08:05,923 calling 'state.apply sys-cacher.install-client'...
2024-06-27 02:08:33,911 output: debian-12-minimal:
2024-06-27 02:08:33,911 output: ----------
2024-06-27 02:08:33,911 output:           ID: sys-cacher-install-client-scripts
2024-06-27 02:08:33,911 output:     Function: file.recurse
2024-06-27 02:08:33,911 output:         Name: /usr/bin/
2024-06-27 02:08:33,912 output:       Result: False
2024-06-27 02:08:33,912 output:      Comment: Recurse failed: none of the specified sources were found
2024-06-27 02:08:33,912 output:      Started: 02:08:33.068632
2024-06-27 02:08:33,912 output:     Duration: 31.698 ms
2024-06-27 02:08:33,912 output:      Changes:   
2024-06-27 02:08:33,912 output: ----------
2024-06-27 02:08:33,912 output:           ID: sys-cacher-install-client-repository-definitions
2024-06-27 02:08:33,912 output:     Function: cmd.run
2024-06-27 02:08:33,912 output:         Name: apt-cacher-ng-repo
2024-06-27 02:08:33,912 output:       Result: False
2024-06-27 02:08:33,912 output:      Comment: One or more requisite failed: sys-cacher.install-client.sys-cacher-install-client-scripts
2024-06-27 02:08:33,912 output:      Started: 02:08:33.103109
2024-06-27 02:08:33,912 output:     Duration: 0.004 ms
2024-06-27 02:08:33,912 output:      Changes:   
2024-06-27 02:08:33,912 output: ----------
2024-06-27 02:08:33,912 output:           ID: sys-cacher-install-client-systemd
2024-06-27 02:08:33,912 output:     Function: file.managed
2024-06-27 02:08:33,912 output:         Name: /usr/lib/systemd/system/qusal-apt-cacher-ng-repo.service
2024-06-27 02:08:33,912 output:       Result: True
2024-06-27 02:08:33,912 output:      Comment: File /usr/lib/systemd/system/qusal-apt-cacher-ng-repo.service is in the correct state
2024-06-27 02:08:33,912 output:      Started: 02:08:33.103203
2024-06-27 02:08:33,912 output:     Duration: 3.896 ms
2024-06-27 02:08:33,912 output:      Changes:   
2024-06-27 02:08:33,912 output: ----------
2024-06-27 02:08:33,912 output:           ID: sys-cacher-install-client-systemd-start-qusal-apt-cacher-ng-repo.service
2024-06-27 02:08:33,913 output:     Function: service.enabled
2024-06-27 02:08:33,913 output:         Name: qusal-apt-cacher-ng-repo.service
2024-06-27 02:08:33,913 output:       Result: True
2024-06-27 02:08:33,913 output:      Comment: Service qusal-apt-cacher-ng-repo.service is already enabled, and is in the desired state
2024-06-27 02:08:33,913 output:      Started: 02:08:33.139695
2024-06-27 02:08:33,913 output:     Duration: 22.963 ms
2024-06-27 02:08:33,913 output:      Changes:   
2024-06-27 02:08:33,913 output: 
2024-06-27 02:08:33,913 output: Summary for debian-12-minimal
2024-06-27 02:08:33,913 output: ------------
2024-06-27 02:08:33,913 output: Succeeded: 2
2024-06-27 02:08:33,913 output: Failed:    2
2024-06-27 02:08:33,913 output: ------------
2024-06-27 02:08:33,913 output: Total states run:     4
2024-06-27 02:08:33,913 output: Total run time:  58.561 ms
2024-06-27 02:08:33,913 exit code: 20
ben-grande commented 1 week ago

The error you are encountering due to a buggy Salt version that Qubes still uses.

I've updated qvm-mgmt. Can you please reinstall it?

sudo qubesctl state.apply dom0.helpers

My output:

$ qvm-mgmt debian-12-minimal
GLOBAL
management_dispvm: dvm-mgmt
management_dispvm template: tpl-mgmt
management_dispvm template features:
  os-distribution: fedora
  os-version: 40
  template-name: fedora-40-minimal
  template-release: 202404260902

WANTED
qube: debian-12-minimal
class: TemplateVM
  template-name: debian-12-minimal
  template-release: 202308031621
  os-distribution: debian
  os-version: 12
debian-12-minimal management_dispvm: dvm-mgmt
$ qctlus debian-12-minimal sys-cacher.install-client
debian-12-minimal:
  ----------
            ID: sys-cacher-install-client-scripts
      Function: file.recurse
          Name: /usr/bin/
        Result: True
       Comment: Recursively updated /usr/bin/
       Started: 00:00:00.00000
      Duration: 106.338 ms
       Changes:
                ----------
                /usr/bin/apt-cacher-ng-repo:
                    ----------
                    diff:
                        ---
                        +++
                        @@ -1,7 +1,7 @@
                        DIFF REDACTED FOR BREVITY
  ----------
            ID: sys-cacher-install-client-repository-definitions
      Function: cmd.run
          Name: apt-cacher-ng-repo
        Result: True
       Comment: configuration remained untouched
       Started: 00:00:00.00000
      Duration: 53.172 ms
       Changes:
  ----------
            ID: sys-cacher-install-client-systemd
      Function: file.managed
          Name: /usr/lib/systemd/system/qusal-apt-cacher-ng-repo.service
        Result: True
       Comment: File /usr/lib/systemd/system/qusal-apt-cacher-ng-repo.service updated
       Started: 00:00:00.00000
      Duration: 9.882 ms
       Changes:
                ----------
                diff:
                    New file
                mode:
                    0644
  ----------
            ID: sys-cacher-install-client-systemd-start-qusal-apt-cacher-ng-repo.service
      Function: service.enabled
          Name: qusal-apt-cacher-ng-repo.service
        Result: True
       Comment: Service qusal-apt-cacher-ng-repo.service has been enabled, and is in the desired state
       Started: 00:00:00.00000
      Duration: 528.199 ms
       Changes:
                ----------
                qusal-apt-cacher-ng-repo.service:
                    True

  Summary for debian-12-minimal
  ------------
  Succeeded: 4 (changed=3)
  Failed:    0
  ------------
  Total states run:     4
  Total run time: 697.591 ms
  /usr/lib/python3.12/site-packages/salt/utils/jid.py:19: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    return datetime.datetime.utcnow()

On tpl-mgmt, my salt version reports 3006-8.1, what does your version reports?

dnf list --installed salt*
Installed Packages
salt.noarch                          3006.8-1.fc40                      @updates
salt-ssh.noarch                      3006.8-1.fc40                      @update

I suppose it is the same version because you just installed the mgmt qube, but just to be sure. If it is the same version and the error still occurs, I don't know what to do but wait for QubesOS to update the Salt package. If it is not the same version, please update tpl-mgmt packages.

c0mmando commented 6 days ago

after reinstalling tpl-mgmt the issue resolved itself!! thanks for the suggestions and guidance.