503 on ansible installation fails to install formula

[radek-otee]

Commitment

I confirm that I have read the following resources:

• How to troubleshoot Qusal
• How to ask questions The Smart Way
• Writing the perfect question
• Question checklist
• Could you please make my preference the default?

Software version

Qubes OS 4.2.1 with sys-cacher installed (this looks like a cacher issue - will investigate further).

Brief summary

Ansible install formula fails with 503

Steps to reproduce

$ sudo qubesctl state.apply ansible.create
$ sudo qubesctl --skip-dom0 --targets=tpl-ansible state.apply ansible.install

  ----------
            ID: common-updated
      Function: pkg.uptodate
        Result: False
       Comment: E: Failed to fetch http://HTTPS///ppa.launchpad.net/ansible/ansible/ubuntu/dists/noble/InRelease  503  SSL error: certificate verify failed [IP: 127.0.0.1 8082]
                E: Some index files failed to download. They have been ignored, or old ones used instead.
       Started: 09:35:59.710702
      Duration: 9492.799 ms
       Changes:

Expected behavior

Expected the installation to go through as normal.

Actual behavior

The script above throws the error above. Full log:

tpl-ansible:
  ----------
            ID: ansible-install-ansible-keyring
      Function: file.managed
          Name: /usr/share/keyrings/ansible.asc
        Result: True
       Comment: File /usr/share/keyrings/ansible.asc is in the correct state
       Started: 09:35:57.340794
      Duration: 41.578 ms
       Changes:   
  ----------
            ID: ansible-remove-ansible-old-format
      Function: file.absent
          Name: /etc/apt/sources.list.d/ansible.list
        Result: True
       Comment: File /etc/apt/sources.list.d/ansible.list is not present
       Started: 09:35:57.382684
      Duration: 0.404 ms
       Changes:   
  ----------
            ID: ansible-install-ansible-repository
      Function: file.managed
          Name: /etc/apt/sources.list.d/ansible.sources
        Result: True
       Comment: File /etc/apt/sources.list.d/ansible.sources updated
       Started: 09:35:57.383364
      Duration: 5.977 ms
       Changes:   
                ----------
                diff:
                    --- 
                    +++ 
                    @@ -1,5 +1,5 @@
                     Types: deb
                    -URIs: http://HTTPS///ppa.launchpad.net/ansible/ansible/ubuntu
                    +URIs: https://ppa.launchpad.net/ansible/ansible/ubuntu
                     Suites: noble
                     Components: main
                     Signed-by: /usr/share/keyrings/ansible.asc
  ----------
            ID: ansible-run-apt-cacher-ng-repo
      Function: cmd.run
          Name: apt-cacher-ng-repo
        Result: True
       Comment: Command "apt-cacher-ng-repo" run
       Started: 09:35:57.391928
      Duration: 16.71 ms
       Changes:   
                ----------
                pid:
                    1201
                retcode:
                    0
                stderr:
                stdout:

                    changed=yes comment='configuration was modified'
  ----------
            ID: common-updated
      Function: pkg.uptodate
        Result: False
       Comment: E: Failed to fetch http://HTTPS///ppa.launchpad.net/ansible/ansible/ubuntu/dists/noble/InRelease  503  SSL error: certificate verify failed [IP: 127.0.0.1 8082]
                E: Some index files failed to download. They have been ignored, or old ones used instead.
       Started: 09:35:59.710702
      Duration: 9492.799 ms
       Changes:   
  ----------
            ID: utils.tools.zsh-touch-home-zshrc
      Function: file.touch
          Name: /home/user/.zshrc
        Result: True
       Comment: Updated times on file /home/user/.zshrc
       Started: 09:36:09.204639
      Duration: 1.331 ms
       Changes:   
                ----------
                touched:
                    /home/user/.zshrc
  ----------
            ID: utils.tools.zsh-touch-skel-zshrc
      Function: file.touch
          Name: /etc/skel/.zshrc
        Result: True
       Comment: Updated times on file /etc/skel/.zshrc
       Started: 09:36:09.206031
      Duration: 1.147 ms
       Changes:   
                ----------
                touched:
                    /etc/skel/.zshrc
  ----------
            ID: utils.tools.zsh-installed
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: utils.tools.common.update.common-updated
       Started: 09:36:09.215817
      Duration: 0.005 ms
       Changes:   
  ----------
            ID: utils.tools.zsh-change-user-shell-to-zsh
      Function: cmd.run
          Name: usermod -s /bin/zsh user
        Result: False
       Comment: One or more requisite failed: utils.tools.zsh.install.utils.tools.zsh-installed
       Started: 09:36:09.216018
      Duration: 0.003 ms
       Changes:   
  ----------
            ID: remove-/home/user/Desktop
      Function: file.absent
          Name: /home/user/Desktop
        Result: True
       Comment: File /home/user/Desktop is not present
       Started: 09:36:09.216073
      Duration: 969.315 ms
       Changes:   
  ----------
            ID: remove-/home/user/Documents
      Function: file.absent
          Name: /home/user/Documents
        Result: True
       Comment: File /home/user/Documents is not present
       Started: 09:36:10.185466
      Duration: 8.581 ms
       Changes:   
  ----------
            ID: remove-/home/user/Downloads
      Function: file.absent
          Name: /home/user/Downloads
        Result: True
       Comment: File /home/user/Downloads is not present
       Started: 09:36:10.194113
      Duration: 8.428 ms
       Changes:   
  ----------
            ID: remove-/home/user/Music
      Function: file.absent
          Name: /home/user/Music
        Result: True
       Comment: File /home/user/Music is not present
       Started: 09:36:10.202602
      Duration: 8.188 ms
       Changes:   
  ----------
            ID: remove-/home/user/Pictures
      Function: file.absent
          Name: /home/user/Pictures
        Result: True
       Comment: File /home/user/Pictures is not present
       Started: 09:36:10.210858
      Duration: 8.021 ms
       Changes:   
  ----------
            ID: remove-/home/user/Public
      Function: file.absent
          Name: /home/user/Public
        Result: True
       Comment: File /home/user/Public is not present
       Started: 09:36:10.218939
      Duration: 8.128 ms
       Changes:   
  ----------
            ID: remove-/home/user/Templates
      Function: file.absent
          Name: /home/user/Templates
        Result: True
       Comment: File /home/user/Templates is not present
       Started: 09:36:10.227127
      Duration: 8.43 ms
       Changes:   
  ----------
            ID: remove-/home/user/Videos
      Function: file.absent
          Name: /home/user/Videos
        Result: True
       Comment: File /home/user/Videos is not present
       Started: 09:36:10.235620
      Duration: 7.95 ms
       Changes:   
  ----------
            ID: dotfiles-copy-sh-home
      Function: file.recurse
          Name: /home/user/
        Result: True
       Comment: Recursively updated /home/user/
       Started: 09:36:10.243631
      Duration: 121.986 ms
       Changes:   
                ----------
                /home/user:
                    ----------
                    /home/user:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /home/user/.config/sh/profile.d:
                    ----------
                    /home/user/.config/sh/profile.d:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /home/user/.local/bin:
                    ----------
                    /home/user/.local/bin:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /home/user/.local/bin/get-script-dir:
                    ----------
                    mode:
                        0644
                /home/user/.local/bin/gpg-sign-email:
                    ----------
                    mode:
                        0644
                /home/user/.local/bin/has:
                    ----------
                    mode:
                        0644
                /home/user/.local/bin/requires-root:
                    ----------
                    mode:
                        0644
                /home/user/.local/bin/resize-terminal:
                    ----------
                    mode:
                        0644
  ----------
            ID: dotfiles-fix-executables-sh-dir-home
      Function: file.directory
          Name: /home/user/.local/bin
        Result: True
       Comment: Directory /home/user/.local/bin updated
       Started: 09:36:10.369382
      Duration: 5.021 ms
       Changes:   
                ----------
                /home/user/.local/bin:
                    ----------
                    mode:
                        0755
                /home/user/.local/bin/get-script-dir:
                    ----------
                    mode:
                        0755
                /home/user/.local/bin/gpg-sign-email:
                    ----------
                    mode:
                        0755
                /home/user/.local/bin/has:
                    ----------
                    mode:
                        0755
                /home/user/.local/bin/requires-root:
                    ----------
                    mode:
                        0755
                /home/user/.local/bin/resize-terminal:
                    ----------
                    mode:
                        0755
                mode:
                    0755
  ----------
            ID: dotfiles-copy-sh-skel
      Function: file.recurse
          Name: /etc/skel
        Result: True
       Comment: Recursively updated /etc/skel
       Started: 09:36:10.374463
      Duration: 77.881 ms
       Changes:   
                ----------
                /etc/skel:
                    ----------
                    /etc/skel:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /etc/skel/.config/sh/profile.d:
                    ----------
                    /etc/skel/.config/sh/profile.d:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /etc/skel/.local/bin:
                    ----------
                    /etc/skel/.local/bin:
                        ----------
                        mode:
                            0700
                    mode:
                        0700
                /etc/skel/.local/bin/get-script-dir:
                    ----------
                    mode:
                        0644
                /etc/skel/.local/bin/gpg-sign-email:
                    ----------
                    mode:
                        0644
                /etc/skel/.local/bin/has:
                    ----------
                    mode:
                        0644
                /etc/skel/.local/bin/requires-root:
                    ----------
                    mode:
                        0644
                /etc/skel/.local/bin/resize-terminal:
                    ----------
                    mode:
                        0644
  ----------
            ID: dotfiles-fix-executables-sh-dir-skel
      Function: file.directory
          Name: /etc/skel/.local/bin
        Result: True
       Comment: Directory /etc/skel/.local/bin updated
       Started: 09:36:10.453863
      Duration: 3.536 ms
       Changes:   
                ----------
                /etc/skel/.local/bin:
                    ----------
                    mode:
                        0755
                /etc/skel/.local/bin/get-script-dir:
                    ----------
                    mode:
                        0755
                /etc/skel/.local/bin/gpg-sign-email:
                    ----------
                    mode:
                        0755
                /etc/skel/.local/bin/has:
                    ----------
                    mode:
                        0755
                /etc/skel/.local/bin/requires-root:
                    ----------
                    mode:
                        0755
                /etc/skel/.local/bin/resize-terminal:
                    ----------
                    mode:
                        0755
                mode:
                    0755
  ----------
            ID: dotfiles-copy-ssh-home
      Function: file.recurse
          Name: /home/user/
        Result: True
       Comment: The directory /home/user/ is in the correct state
       Started: 09:36:10.457453
      Duration: 15.425 ms
       Changes:   
  ----------
            ID: dotfiles-copy-ssh-skel
      Function: file.recurse
          Name: /etc/skel/
        Result: True
       Comment: The directory /etc/skel/ is in the correct state
       Started: 09:36:10.472950
      Duration: 8.041 ms
       Changes:   
  ----------
            ID: dotfiles-copy-x11-home
      Function: file.recurse
          Name: /home/user/
        Result: True
       Comment: Recursively updated /home/user/
       Started: 09:36:10.481053
      Duration: 63.081 ms
       Changes:   
                ----------
                /home/user:
                    ----------
                    /home/user:
                        ----------
                        mode:
                            0755
                    mode:
                        0755
                /home/user/.config/sh/profile.d:
                    ----------
                    /home/user/.config/sh/profile.d:
                        ----------
                        mode:
                            0755
                    mode:
                        0755
  ----------
            ID: dotfiles-copy-x11-skel
      Function: file.recurse
          Name: /etc/skel/
        Result: True
       Comment: Recursively updated /etc/skel/
       Started: 09:36:10.545758
      Duration: 53.487 ms
       Changes:   
                ----------
                /etc/skel:
                    ----------
                    /etc/skel:
                        ----------
                        mode:
                            0755
                    mode:
                        0755
                /etc/skel/.config/sh/profile.d:
                    ----------
                    /etc/skel/.config/sh/profile.d:
                        ----------
                        mode:
                            0755
                    mode:
                        0755
  ----------
            ID: sys-ssh-agent-client-installed
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: utils.tools.common.update.common-updated
       Started: 09:36:10.605418
      Duration: 0.005 ms
       Changes:   
  ----------
            ID: sys-ssh-agent-client-installed-os-specific
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: utils.tools.common.update.common-updated
       Started: 09:36:10.605508
      Duration: 0.002 ms
       Changes:   
  ----------
            ID: sys-ssh-agent-client-system-systemd-dir
      Function: file.recurse
          Name: /usr/lib/systemd/system/
        Result: True
       Comment: The directory /usr/lib/systemd/system/ is in the correct state
       Started: 09:36:10.606594
      Duration: 7.776 ms
       Changes:   
  ----------
            ID: ansible-installed
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: utils.tools.common.update.common-updated
       Started: 09:36:10.614632
      Duration: 0.003 ms
       Changes:   
  ----------
            ID: ansible-installed-os-specific
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: utils.tools.common.update.common-updated
       Started: 09:36:10.614743
      Duration: 0.002 ms
       Changes:   
  ----------
            ID: ansible-ssh-config
      Function: file.managed
          Name: /etc/ssh/ssh_config.d/99-ssh-ansible.conf
        Result: False
       Comment: Parent directory not present
       Started: 09:36:10.614778
      Duration: 2.586 ms
       Changes:   
  ----------
            ID: ansible-sshd-config
      Function: file.managed
          Name: /etc/ssh/sshd_config.d/99-sshd-ansible.conf
        Result: False
       Comment: Parent directory not present
       Started: 09:36:10.617416
      Duration: 2.628 ms
       Changes:   

  Summary for tpl-ansible
  -------------
  Succeeded: 23 (changed=10)
  Failed:     9
  -------------
  Total states run:     32
  Total run time:   10.948 s
  /usr/lib/python3.12/site-packages/salt/utils/jid.py:19: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    return datetime.datetime.utcnow()

.

[ben-grande]

Failed to fetch http://HTTPS///ppa.launchpad.net/ansible/ansible/ubuntu/dists/noble/InRelease  503  SSL error: certificate verify failed [IP: 127.0.0.1 8082]

503 is a server error. The certificate of Launchpad is invalid: https://ppa.launchpad.net/

It is an error on their side, I will keep this issue open until they fix the certificate error.

[ben-grande]

Another possibility is that they never had https on the ppa.launchpad.net and I did a mistake when editing the repository sources by adding https. See Askubuntu ppa.launchpad.net does not support https. I will try it later today and switch to the URL that supports HTTPS.

[ben-grande]

ppa.launchpad.net never supported HTTPS. Changed was made to all repositories to support HTTPS after cacher support in b2c9479e50c1d19468e0fdd9ac9dfb65f322146c.

The host https://ppa.launchpadcontent.net does support HTTPS and the repository signing key is the same, so no trouble with tampering. Upstream discussion.