search

coreos / zincati

Agent for Fedora CoreOS auto-updates
https://coreos.github.io/zincati
Apache License 2.0
156 stars 33 forks source link

Unable to trigger a manual update with rpm-ostree when a zincati update strategy is active #1072

Closed kavishgr closed 1 year ago

kavishgr commented 1 year ago

I am experiencing an issue with rpm-ostree while trying to manually update the system. Currently, I have configured my zincati to perform updates automatically on weekends. However, when I check for updates with rpm-ostree, it notifies me that there is an update pending:

❯ rpm-ostree status -v
State: idle
AutomaticUpdatesDriver: Zincati (zincati.service)
  DriverState: active; update staged: 38.20230709.3.0; reboot pending due to update strategy
Deployments:
  fedora:fedora/x86_64/coreos/stable (index: 0)
                  Version: 38.20230709.3.0 (2023-07-24T12:25:01Z)
                   Commit: 552de26fe0fe6a5e491f7a4163db125e3d44b144ae53a8f5f488e3f8481c46f9
                           `- fedora-coreos-pool (2023-07-22T22:42:27Z)
                   Staged: yes
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Mon Jul 24 16:27:47 2023 using RSA key ID 809A8D7CEB10B464
                           Good signature from "Fedora <fedora-38-primary@fedoraproject.org>"
                 Upgraded: NetworkManager 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-cloud-setup 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-libnm 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-team 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-tui 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           aardvark-dns 1.6.0-1.fc38 -> 1.7.0-1.fc38
                           amd-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           atheros-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           brcmfmac-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           container-selinux 2:2.218.0-1.fc38 -> 2:2.219.0-1.fc38
                           diffutils 3.9-1.fc38 -> 3.10-1.fc38
                           elfutils-default-yama-scope 0.189-2.fc38 -> 0.189-3.fc38
                           elfutils-libelf 0.189-2.fc38 -> 0.189-3.fc38
                           elfutils-libs 0.189-2.fc38 -> 0.189-3.fc38
                           ethtool 2:6.3-1.fc38 -> 2:6.4-1.fc38
                           intel-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           kernel 6.3.8-200.fc38 -> 6.3.11-200.fc38
                           kernel-core 6.3.8-200.fc38 -> 6.3.11-200.fc38
                           kernel-modules 6.3.8-200.fc38 -> 6.3.11-200.fc38
                           kernel-modules-core 6.3.8-200.fc38 -> 6.3.11-200.fc38
                           libassuan 2.5.5-6.fc38 -> 2.5.6-1.fc38
                           libipa_hbac 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           libksba 1.6.3-2.fc38 -> 1.6.4-1.fc38
                           libnetfilter_conntrack 1.0.8-7.fc38 -> 1.0.9-1.fc38
                           libreport-filesystem 2.17.10-1.fc38 -> 2.17.11-1.fc38
                           libsmbclient 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                           libsss_certmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           libsss_idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           libsss_nss_idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           libsss_sudo 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           libwbclient 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                           libxcrypt 4.4.35-1.fc38 -> 4.4.36-1.fc38
                           linux-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           linux-firmware-whence 20230515-150.fc38 -> 20230625-151.fc38
                           mt7xxx-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           netavark 1.6.0-2.fc38 -> 1.7.0-1.fc38
                           nvidia-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           ostree 2023.4-1.fc38 -> 2023.4-2.fc38
                           ostree-libs 2023.4-1.fc38 -> 2023.4-2.fc38
                           passt 0^20230509.g96f8d55-1.fc38 -> 0^20230625.g32660ce-1.fc38
                           passt-selinux 0^20230509.g96f8d55-1.fc38 -> 0^20230625.g32660ce-1.fc38
                           realtek-firmware 20230515-150.fc38 -> 20230625-151.fc38
                           rpm-ostree 2023.4-2.fc38 -> 2023.5-1.fc38
                           rpm-ostree-libs 2023.4-2.fc38 -> 2023.5-1.fc38
                           rpm-sequoia 1.4.0-3.fc38 -> 1.4.1-1.fc38
                           samba-client-libs 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                           samba-common 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                           samba-common-libs 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                           selinux-policy 38.17-1.fc38 -> 38.20-1.fc38
                           selinux-policy-targeted 38.17-1.fc38 -> 38.20-1.fc38
                           sssd-ad 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-client 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-common 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-common-pac 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-ipa 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-krb5 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-krb5-common 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-ldap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           sssd-nfs-idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                           vim-data 2:9.0.1649-1.fc38 -> 2:9.0.1671-1.fc38
                           vim-minimal 2:9.0.1649-1.fc38 -> 2:9.0.1671-1.fc38

* fedora:fedora/x86_64/coreos/stable (index: 1)
                  Version: 38.20230625.3.0 (2023-07-11T11:57:53Z)
                   Commit: e841d77aadb875bb801ac845a0d9b8a70b4224bdeb15e7d6c5bff1da932c0301
                           `- fedora-coreos-pool (2023-07-09T08:04:53Z)
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Tue Jul 11 15:59:48 2023 using RSA key ID 809A8D7CEB10B464
                           Good signature from "Fedora <fedora-38-primary@fedoraproject.org>"

  fedora:fedora/x86_64/coreos/stable (index: 2)
                  Version: 38.20230527.3.0 (2023-06-13T00:08:09Z)
                   Commit: a13f4a93e6ce9ba5b0a00f74dc68ee3794aa6910c523aa6b5572e3669b8c1d7e
                           `- fedora-coreos-pool (2023-06-09T22:45:29Z)
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Tue Jun 13 04:10:04 2023 using RSA key ID 809A8D7CEB10B464
                           Good signature from "Fedora <fedora-38-primary@fedoraproject.org>"

AvailableUpdate:
        Version: 38.20230709.3.0 (2023-07-24T12:25:01Z)
         Commit: 552de26fe0fe6a5e491f7a4163db125e3d44b144ae53a8f5f488e3f8481c46f9
   GPGSignature: 1 signature
                 Signature made Mon Jul 24 16:27:47 2023 using RSA key ID 809A8D7CEB10B464
                 Good signature from "Fedora <fedora-38-primary@fedoraproject.org>"
       Upgraded: NetworkManager 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                 NetworkManager-cloud-setup 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                 NetworkManager-libnm 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                 NetworkManager-team 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                 NetworkManager-tui 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                 aardvark-dns 1.6.0-1.fc38 -> 1.7.0-1.fc38
                 amd-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 atheros-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 brcmfmac-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 container-selinux 2:2.218.0-1.fc38 -> 2:2.219.0-1.fc38
                 diffutils 3.9-1.fc38 -> 3.10-1.fc38
                 elfutils-default-yama-scope 0.189-2.fc38 -> 0.189-3.fc38
                 elfutils-libelf 0.189-2.fc38 -> 0.189-3.fc38
                 elfutils-libs 0.189-2.fc38 -> 0.189-3.fc38
                 ethtool 2:6.3-1.fc38 -> 2:6.4-1.fc38
                 intel-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 kernel 6.3.8-200.fc38 -> 6.3.11-200.fc38
                 kernel-core 6.3.8-200.fc38 -> 6.3.11-200.fc38
                 kernel-modules 6.3.8-200.fc38 -> 6.3.11-200.fc38
                 kernel-modules-core 6.3.8-200.fc38 -> 6.3.11-200.fc38
                 libassuan 2.5.5-6.fc38 -> 2.5.6-1.fc38
                 libipa_hbac 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 libksba 1.6.3-2.fc38 -> 1.6.4-1.fc38
                 libnetfilter_conntrack 1.0.8-7.fc38 -> 1.0.9-1.fc38
                 libreport-filesystem 2.17.10-1.fc38 -> 2.17.11-1.fc38
                 libsmbclient 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                 libsss_certmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 libsss_idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 libsss_nss_idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 libsss_sudo 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 libwbclient 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                 libxcrypt 4.4.35-1.fc38 -> 4.4.36-1.fc38
                 linux-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 linux-firmware-whence 20230515-150.fc38 -> 20230625-151.fc38
                 mt7xxx-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 netavark 1.6.0-2.fc38 -> 1.7.0-1.fc38
                 nvidia-gpu-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 ostree 2023.4-1.fc38 -> 2023.4-2.fc38
                 ostree-libs 2023.4-1.fc38 -> 2023.4-2.fc38
                 passt 0^20230509.g96f8d55-1.fc38 -> 0^20230625.g32660ce-1.fc38
                 passt-selinux 0^20230509.g96f8d55-1.fc38 -> 0^20230625.g32660ce-1.fc38
                 realtek-firmware 20230515-150.fc38 -> 20230625-151.fc38
                 rpm-ostree 2023.4-2.fc38 -> 2023.5-1.fc38
                 rpm-ostree-libs 2023.4-2.fc38 -> 2023.5-1.fc38
                 rpm-sequoia 1.4.0-3.fc38 -> 1.4.1-1.fc38
                 samba-client-libs 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                 samba-common 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                 samba-common-libs 2:4.18.3-3.fc38 -> 2:4.18.4-0.fc38
                 selinux-policy 38.17-1.fc38 -> 38.20-1.fc38
                 selinux-policy-targeted 38.17-1.fc38 -> 38.20-1.fc38
                 sssd-ad 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-client 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-common 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-common-pac 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-ipa 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-krb5 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-krb5-common 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-ldap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 sssd-nfs-idmap 2.9.0-1.fc38 -> 2.9.1-1.fc38
                 vim-data 2:9.0.1649-1.fc38 -> 2:9.0.1671-1.fc38
                 vim-minimal 2:9.0.1649-1.fc38 -> 2:9.0.1671-1.fc38

but attempting to upgrade results in a message stating that no upgrade is available:

❯ sudo rpm-ostree upgrade --bypass-driver
⠤ Receiving metadata objects: 1/(estimating) 65 bytes/s 196 bytes...       
2 metadata, 0 content objects fetched; 
788 B transferred in 6 seconds; 0 bytes content written
Receiving metadata objects: 1/(estimating) 65 bytes/s 196 bytes... done
No upgrade available.

This is my update strategy:

❯ cat 90-updates-strategy.toml
[updates]
strategy = "periodic"

[updates.periodic]
time_zone = "Indian/Mauritius"

[[updates.periodic.window]]
days = [ "Sat", "Sun" ]
start_time = "10:00"
length_minutes = 60

Ideally, I should be able to update the system manually regardless of the zincati update strategy. However, this is not happening. The only way I can trigger a manual update is by disabling the zincati configuration by renaming it:

❯ sudo mv 90-updates-strategy.toml 90-updates-strategy.toml.NOT-NOW

and restarting the zincati service:

❯ sudo systemctl restart zincati

When I check with the rpm-ostree status -v command, the state shows as busy, indicating that it is about to update to a new deployment:

❯ rpm-ostree status -v
State: busy
AutomaticUpdatesDriver: Zincati (zincati.service)
  DriverState: active; found update on remote: 38.20230709.3.0
Transaction: deploy --lock-finalization --skip-branch-check revision=552de26fe0fe6a5e491f7a4163db125e3d44b144ae53a8f5f488e3f8481c46f9 --disallow-downgrade
  Initiator: caller :1.109
Deployments:
  fedora:fedora/x86_64/coreos/stable (index: 0)
                  Version: 38.20230709.3.0 (2023-07-24T12:25:01Z)
                   Commit: 552de26fe0fe6a5e491f7a4163db125e3d44b144ae53a8f5f488e3f8481c46f9
                           `- fedora-coreos-pool (2023-07-22T22:42:27Z)
                   Staged: yes
                StateRoot: fedora-coreos
             GPGSignature: 1 signature
                           Signature made Mon Jul 24 16:27:47 2023 using RSA key ID 809A8D7CEB10B464
                           Good signature from "Fedora <fedora-38-primary@fedoraproject.org>"
                 Upgraded: NetworkManager 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-cloud-setup 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-libnm 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
                           NetworkManager-team 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38

                           ---- REDACTED ----

Following this, I receive a broadcast message notifying me that an update is about to take place:

Broadcast message from Zincati at Thu 2023-07-27 14:07:36 UTC:
New update 38.20230709.3.0 is available and has been deployed.
If permitted by the update strategy, Zincati will reboot into this update when
all interactive users have logged out, or in 10 minutes, whichever comes
earlier. Please log out of all active sessions in order to let the auto-update
process continue.

This behavior is unexpected. I guess I should be able to update the system at any time without having to alter the zincati configuration.

cgwalters commented 1 year ago

Yes, this is extremely confusing. I generally agree with you, however there's a high degree of subtlety and complexity in the interaction between the two components today, and "user control" becomes a major third one.

This also relates to https://github.com/coreos/zincati/issues/928 I think possibly.

And a definite factor that's often bit me in the past is how zincati leaves around the deployment finalization lock even if it's explicitly stopped.

I think as a bottom line today though, if you want to take the wheel, you need to at least temporarily stop zincati...sorry.

dustymabe commented 1 year ago

I think this is basically https://github.com/coreos/zincati/issues/498

cgwalters commented 1 year ago

Yeah looks like it, closing as duplicate