search

coreos / rpm-ostree

⚛📦 Hybrid image/package system with atomic upgrades and package layering
https://coreos.github.io/rpm-ostree
Other
875 stars 196 forks source link

error during finalizing previous deployment #1791

Open dustymabe opened 5 years ago

dustymabe commented 5 years ago

I see the following error after reboot:

[dustymabe@hattop ~]$ sudo rpm-ostree status 
[sudo] password for dustymabe: 
State: idle
Warning: failed to finalize previous deployment
         error: During /etc merge: Reading modified config file: fstatat(cups/ppd/19w202_color.ppd): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
Deployments:
● ostree://home:fedora/29/x86_64/silverblue

The systemd unit log output shows:

$ sudo journalctl -b -1 -u ostree-finalize-staged.service
-- Logs begin at Fri 2019-03-15 09:52:42 EDT, end at Mon 2019-03-18 11:09:58 EDT. --
Mar 18 10:48:55 hattop systemd[1]: Started OSTree Finalize Staged Deployment.
Mar 18 10:57:08 hattop systemd[1]: Stopping OSTree Finalize Staged Deployment...
Mar 18 10:57:08 hattop ostree[9674]: Failed to get connection to xfconfd: Cannot autolaunch D-Bus without X11 $DISPLAY
Mar 18 10:57:08 hattop ostree[9674]: Finalizing staged deployment
Mar 18 10:57:10 hattop ostree[9674]: Copying /etc changes: 28 modified, 0 removed, 139 added
Mar 18 10:57:10 hattop ostree[9674]: error: During /etc merge: Reading modified config file: fstatat(cups/ppd/19w202_color.ppd): No such file or directory
Mar 18 10:57:10 hattop systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited status=1
Mar 18 10:57:10 hattop systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Mar 18 10:57:10 hattop systemd[1]: Stopped OSTree Finalize Staged Deployment.

Host system details

``` [dustymabe@hattop ~]$ sudo rpm-ostree status State: idle Warning: failed to finalize previous deployment error: During /etc merge: Reading modified config file: fstatat(cups/ppd/19w202_color.ppd): No such file or directory check `journalctl -b -1 -u ostree-finalize-staged.service` AutomaticUpdates: disabled Deployments: ● ostree://home:fedora/29/x86_64/silverblue Version: 29.20190216.0 (2019-02-16T00:50:00Z) BaseCommit: 991c687805f00dc902e5ef7650ee1cc57ca474c6a8bb668e2f7f67bcda29a037 GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4 ReplacedBasePackages: NetworkManager-config-connectivity-fedora NetworkManager-bluetooth NetworkManager-adsl NetworkManager-ppp NetworkManager NetworkManager-libnm NetworkManager-wwan NetworkManager-wifi 1:1.12.6-5.fc29 -> 1:1.12.6-1.fc29 LayeredPackages: NetworkManager-iodine-gnome NetworkManager-l2tp-gnome NetworkManager-libreswan-gnome NetworkManager-ovs NetworkManager-sstp-gnome NetworkManager-strongswan-gnome NetworkManager-team abrt-desktop abrt-java-connector atril atril-caja atril-thumbnailer awscli blivet-gui blueberry caja caja-actions caja-image-converter caja-open-terminal caja-sendto caja-wallpaper caja-xattr-tags chromium ctags dconf-editor dnfdragora-updater emoji-picker engrampa eog eom evince exaile f29-backgrounds-extras-base f29-backgrounds-extras-mate f29-backgrounds-mate fedpkg filezilla firewall-config fuse-sshfs gedit genisoimage gimp gimp-heif-plugin git git-annex git-email gnome-epub-thumbnailer gnome-logs gnote gparted gstreamer1-plugins-ugly-free gtk2-engines gucharmap gvfs-nfs gvim hexchat htop i3 i3lock ifuse initial-setup-gui krb5-workstation libheif libimobiledevice-utils libmatekbd libmatemixer libmateweather libreoffice-calc libreoffice-impress libreoffice-writer libvirt-client libvirt-daemon-config-network libvirt-daemon-kvm libvirt-nss lightdm lm_sensors lshw lynx marco mate-applets mate-backgrounds mate-calc mate-control-center mate-desktop mate-dictionary mate-disk-usage-analyzer mate-icon-theme mate-media mate-menus mate-menus-preferences-category-menu mate-notification-daemon mate-panel mate-polkit mate-power-manager mate-screensaver mate-screenshot mate-search-tool mate-session-manager mate-settings-daemon mate-system-log mate-system-monitor mate-terminal mate-themes mate-user-admin mate-user-guide mosh mozo nbd network-manager-applet nextcloud-client nitrogen nmap ntp openldap-clients orca origin-clients p7zip p7zip-plugins parole pass pavucontrol pluma powerline powertop pwgen pykickstart python-bugzilla python3-psutil python3-reportlab qemu-system-aarch64 rdesktop redshift seahorse seahorse-caja setroubleshoot simple-scan slick-greeter-mate speedtest-cli sshuttle strace system-config-language system-config-printer system-config-printer-applet system-config-users thunderbird tig tmux tmux-powerline transmission-gtk vagrant vagrant-libvirt vagrant-sshfs vim vim-enhanced vim-nerdtree vim-plugin-powerline virt-install virt-manager virt-viewer vlc volumeicon xbacklight xcompmgr xfburn LocalPackages: redhat-internal-cert-install-0.1-9.el7.csb.noarch rpmfusion-free-release-29-1.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch ostree://home:fedora/29/x86_64/silverblue Version: 29.20190120.0 (2019-01-20T00:52:12Z) BaseCommit: 60a9643b331204d7a0d04aa2826e34416cb31ccd901c667752dcdd65ba49e49d GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4 ReplacedBasePackages: NetworkManager-config-connectivity-fedora NetworkManager-bluetooth NetworkManager-adsl NetworkManager-ppp NetworkManager NetworkManager-libnm NetworkManager-wwan NetworkManager-wifi 1:1.12.6-5.fc29 -> 1:1.12.6-1.fc29 LayeredPackages: NetworkManager-iodine-gnome NetworkManager-l2tp-gnome NetworkManager-libreswan-gnome NetworkManager-ovs NetworkManager-sstp-gnome NetworkManager-strongswan-gnome NetworkManager-team abrt-desktop abrt-java-connector atril atril-caja atril-thumbnailer awscli blivet-gui blueberry caja caja-actions caja-image-converter caja-open-terminal caja-sendto caja-wallpaper caja-xattr-tags chromium ctags dconf-editor dnfdragora-updater emoji-picker engrampa eog eom evince exaile f29-backgrounds-extras-base f29-backgrounds-extras-mate f29-backgrounds-mate fedpkg filezilla firewall-config fuse-sshfs gedit genisoimage gimp gimp-heif-plugin git git-annex git-email gnome-epub-thumbnailer gnome-logs gnote gparted gstreamer1-plugins-ugly-free gtk2-engines gucharmap gvfs-nfs gvim hexchat htop i3 i3lock ifuse initial-setup-gui krb5-workstation libheif libimobiledevice-utils libmatekbd libmatemixer libmateweather libreoffice-calc libreoffice-impress libreoffice-writer libvirt-client libvirt-daemon-config-network libvirt-daemon-kvm libvirt-nss lightdm lm_sensors lshw lynx marco mate-applets mate-backgrounds mate-calc mate-control-center mate-desktop mate-dictionary mate-disk-usage-analyzer mate-icon-theme mate-media mate-menus mate-menus-preferences-category-menu mate-notification-daemon mate-panel mate-polkit mate-power-manager mate-screensaver mate-screenshot mate-search-tool mate-session-manager mate-settings-daemon mate-system-log mate-system-monitor mate-terminal mate-themes mate-user-admin mate-user-guide mosh mozo nbd network-manager-applet nextcloud-client nitrogen nmap ntp openldap-clients orca origin-clients p7zip p7zip-plugins parole pass pavucontrol pluma powerline powertop pwgen pykickstart python-bugzilla python3-psutil python3-reportlab qemu-system-aarch64 rdesktop redshift seahorse seahorse-caja setroubleshoot simple-scan slick-greeter-mate speedtest-cli sshuttle strace system-config-language system-config-printer system-config-printer-applet system-config-users thunderbird tig tmux tmux-powerline transmission-gtk vagrant vagrant-libvirt vagrant-sshfs vim vim-enhanced vim-nerdtree vim-plugin-powerline virt-install virt-manager virt-viewer vlc volumeicon xbacklight xcompmgr xfburn LocalPackages: redhat-internal-cert-install-0.1-9.el7.csb.noarch rpmfusion-free-release-29-1.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch Pinned: yes ostree://home:fedora/29/x86_64/silverblue Version: 29.20190112.0 (2019-01-12T00:49:46Z) BaseCommit: ad2a133614fb3d72f9e7e11acc8f8a9d246a8112d14d03a455cb094db307f337 GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4 ReplacedBasePackages: NetworkManager-config-connectivity-fedora NetworkManager-bluetooth NetworkManager-adsl NetworkManager-ppp NetworkManager NetworkManager-libnm NetworkManager-wwan NetworkManager-wifi 1:1.12.6-4.fc29 -> 1:1.12.6-1.fc29, fuse-overlayfs 0.1-8.dev.git91bb401.fc29 -> 0.1-6.dev.git3d48bf9.fc29, podman 1:0.12.1.2-1.git9551f6b.fc29 -> 1:0.12.1.1-1.git66d3499.fc29 LayeredPackages: NetworkManager-iodine-gnome NetworkManager-l2tp-gnome NetworkManager-libreswan-gnome NetworkManager-ovs NetworkManager-sstp-gnome NetworkManager-strongswan-gnome NetworkManager-team abrt-desktop abrt-java-connector atril atril-caja atril-thumbnailer awscli blivet-gui blueberry caja caja-actions caja-image-converter caja-open-terminal caja-sendto caja-wallpaper caja-xattr-tags chromium ctags dconf-editor dnfdragora-updater engrampa eog eom evince exaile f29-backgrounds-extras-base f29-backgrounds-extras-mate f29-backgrounds-mate fedpkg filezilla firewall-config fuse-sshfs gedit genisoimage gimp gimp-heif-plugin git git-annex git-email gnome-epub-thumbnailer gnome-logs gnote gparted gstreamer1-plugins-ugly-free gtk2-engines gucharmap gvfs-nfs gvim hexchat htop i3 i3lock ifuse initial-setup-gui krb5-workstation libheif libimobiledevice-utils libmatekbd libmatemixer libmateweather libreoffice-calc libreoffice-impress libreoffice-writer libvirt-client libvirt-daemon-config-network libvirt-daemon-kvm libvirt-nss lightdm lm_sensors lshw lynx marco mate-applets mate-backgrounds mate-calc mate-control-center mate-desktop mate-dictionary mate-disk-usage-analyzer mate-icon-theme mate-media mate-menus mate-menus-preferences-category-menu mate-notification-daemon mate-panel mate-polkit mate-power-manager mate-screensaver mate-screenshot mate-search-tool mate-session-manager mate-settings-daemon mate-system-log mate-system-monitor mate-terminal mate-themes mate-user-admin mate-user-guide mosh mozo nbd network-manager-applet nextcloud-client nitrogen nmap ntp openldap-clients orca origin-clients p7zip p7zip-plugins parole pass pavucontrol pluma powerline powertop pwgen pykickstart python-bugzilla python3-psutil python3-reportlab qemu-system-aarch64 rdesktop redshift seahorse seahorse-caja setroubleshoot simple-scan slick-greeter-mate speedtest-cli sshuttle strace system-config-language system-config-printer system-config-printer-applet system-config-users thunderbird tig tmux tmux-powerline transmission-gtk vagrant vagrant-libvirt vagrant-sshfs vim vim-enhanced vim-nerdtree vim-plugin-powerline virt-install virt-manager virt-viewer vlc volumeicon xbacklight xcompmgr xfburn LocalPackages: redhat-internal-cert-install-0.1-9.el7.csb.noarch rpmfusion-free-release-29-1.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch Pinned: yes ostree://home:fedora/29/x86_64/silverblue Version: 29.20181209.0 (2018-12-09T01:05:12Z) BaseCommit: ab52579b9965a871e3db1987c6e0a256fecaeb03a3d72df1a8b76baaf2406d8d GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4 ReplacedBasePackages: NetworkManager-config-connectivity-fedora NetworkManager-bluetooth NetworkManager-adsl NetworkManager-ppp NetworkManager NetworkManager-libnm NetworkManager-wwan NetworkManager-wifi 1:1.12.4-2.fc29 -> 1:1.12.6-1.fc29, fuse-overlayfs 0.1-5.dev.gitd40ac75.fc29 -> 0.1-6.dev.git3d48bf9.fc29, podman 1:0.10.1.3-4.gitdb08685.fc29 -> 1:0.12.1.1-1.git66d3499.fc29 LayeredPackages: awscli chromium ctags eog evince fedpkg fuse-sshfs gedit genisoimage gimp git git-annex git-email gvim htop i3 i3lock ifuse krb5-workstation libimobiledevice-utils libreoffice-calc libreoffice-impress libreoffice-writer libvirt-client libvirt-daemon-config-network libvirt-daemon-kvm libvirt-nss lshw lynx mosh nbd network-manager-applet nextcloud-client nitrogen nmap ntp openldap-clients origin-clients pass pavucontrol powerline powertop pwgen pykickstart python-bugzilla python3-reportlab qemu-system-aarch64 rdesktop redshift speedtest-cli sshuttle strace thunderbird tig tmux tmux-powerline vagrant vagrant-libvirt vagrant-sshfs vim vim-nerdtree vim-plugin-powerline virt-install virt-manager virt-viewer vlc volumeicon xbacklight xcompmgr LocalPackages: redhat-internal-cert-install-0.1-9.el7.csb.noarch rpmfusion-free-release-29-1.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch Pinned: yes ```
dustymabe commented 5 years ago

just putting this here for now while we investigate.

dustymabe commented 5 years ago

just putting this here for now while we investigate.

cgwalters commented 5 years ago

Mar 18 10:57:08 hattop ostree[9674]: Failed to get connection to xfconfd: Cannot autolaunch D-Bus without X11 $DISPLAY

Uh...what? What the heck is that - looks like ostree trying to launch a session dbus or so?

Mar 18 10:57:10 hattop ostree[9674]: error: During /etc merge: Reading modified config file: fstatat(cups/ppd/19w202_color.ppd): No such file or directory

Yeah there's an earlier version of this, cupsd.service hasn't shut down fully by the time ostree-finalize-staged starts.

cgwalters commented 5 years ago

I think probably what's happening is we're seeing this a lot with cupsd.service because it's only serializing its state on shutdown or so. Or possibly it's getting IPP notifications of local printers and writing them to disk often?

jlebon commented 5 years ago

Just offhand skimming systemctl cat ostree-finalize-staged.service, I'm thinking we could change After=basic.target to After=local-fs.target & Before=basic.target so we finalize even later.

dustymabe commented 5 years ago

I re-ran the rpm-ostree upgrade --install tmuxinator command that I tried in my first attempt and it did succeed on the second try.

vtolstov commented 5 years ago

as i see this is something that i reported some time ago..

dustymabe commented 5 years ago

hit this again today:

[dustymabe@hattop ~]$ sudo rpm-ostree status
[sudo] password for dustymabe: 
State: idle
Warning: failed to finalize previous deployment
         error: During /etc merge: Reading modified config file: fstatat(cups/ppd/test.ppd): No such file or directory
         check `journalctl -b -1 -u ostree-finalize-staged.service`
AutomaticUpdates: disabled
Deployments:
● ostree://home:fedora/29/x86_64/silverblue
                   Version: 29.20190501.0 (2019-05-01T00:40:50Z)
                BaseCommit: 9d45a122c91deb47cb7b0a35aa8434170bb3720cb0a32cb4690828b4dbb8c1dc

[dustymabe@hattop ~]$ rpm -q ostree rpm-ostree
ostree-2019.1-3.fc29.x86_64
rpm-ostree-2019.3-1.fc29.x86_64
[dustymabe@hattop ~]$ sudo journalctl -b -1 -u ostree-finalize-staged.service
-- Logs begin at Sun 2019-05-12 21:21:22 EDT, end at Tue 2019-05-14 10:50:24 EDT. --
May 14 10:41:51 hattop systemd[1]: Started OSTree Finalize Staged Deployment.
May 14 10:45:15 hattop systemd[1]: Stopping OSTree Finalize Staged Deployment...
May 14 10:45:15 hattop ostree[31392]: Failed to get connection to xfconfd: Cannot autolaunch D-Bus without X11 $DISPLAY
May 14 10:45:15 hattop ostree[31392]: Finalizing staged deployment
May 14 10:45:16 hattop ostree[31392]: Copying /etc changes: 29 modified, 0 removed, 80 added
May 14 10:45:16 hattop ostree[31392]: error: During /etc merge: Reading modified config file: fstatat(cups/ppd/test.ppd): No such file or directory
May 14 10:45:16 hattop systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited status=1
May 14 10:45:16 hattop systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
May 14 10:45:16 hattop systemd[1]: Stopped OSTree Finalize Staged Deployment.
jlebon commented 5 years ago

This should already be fixed in FSB 30 by https://github.com/ostreedev/ostree/pull/1840.