search

coreos / rpm-ostree

⚛📦 Hybrid image/package system with atomic upgrades and package layering
https://coreos.github.io/rpm-ostree
Other
858 stars 195 forks source link

Issue with btrfs compression #2761

Closed LyesSaadi closed 3 years ago

LyesSaadi commented 3 years ago

Host system details

State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.service: last run failed
Transaction: upgrade 
  Initiator: client(id:cli dbus:1.996 unit:vte-spawn-22882dd5-10b3-41df-9b8b-34da2e890714.scope uid:1000)
Deployments:
● ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210415.n.0 (2021-04-15T08:06:47Z)
                BaseCommit: 24172968655aa4322456f9c7012abb275e95b271fe9da931a6c6c7be8d217cde
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: bat brightnessctl broadcom-wl chromium cronie dconf-editor dialect epiphany ffmpeg file-roller firefox-nightly fish gcc
                            gnome-shell-extension-gsconnect gnome-tweaks gnome-weather grim gstreamer1-libav gstreamer1-plugin-openh264 gstreamer1-plugins-bad-freeworld
                            gstreamer1-plugins-base-tools gstreamer1-plugins-good-extras gstreamer1-plugins-ugly gstreamer1-vaapi hunspell-en hunspell-fr hunspell-it kmodtool
                            langpacks-fr lxpolkit mako man-pages-fr mbpfan minecraft-launcher mozilla-openh264 mpv nautilus-gsconnect neovim network-manager-applet npm nushell
                            power-profiles-daemon rpmfusion-free-release rpmfusion-nonfree-release setzer slurp starship swayidle swaylock thunderbird vlc waybar wireguard-tools
                            wl-clipboard wlogout wmctrl wofi
             LocalPackages: cage-0.1.2.1-4.fc34.x86_64 wlroots-0.13.0-1.fc34.x86_64 sway-1.6-1.fc34.x86_64
                  Unlocked: hotfix

  ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210415.n.0 (2021-04-15T08:06:47Z)
                BaseCommit: 24172968655aa4322456f9c7012abb275e95b271fe9da931a6c6c7be8d217cde
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: bat brightnessctl broadcom-wl chromium cronie dconf-editor dialect epiphany ffmpeg file-roller firefox-nightly fish gcc
                            gnome-shell-extension-gsconnect gnome-tweaks gnome-weather grim gstreamer1-libav gstreamer1-plugin-openh264 gstreamer1-plugins-bad-freeworld
                            gstreamer1-plugins-base-tools gstreamer1-plugins-good-extras gstreamer1-plugins-ugly gstreamer1-vaapi hunspell-en hunspell-fr hunspell-it kmodtool
                            langpacks-fr lxpolkit mako man-pages-fr mbpfan minecraft-launcher mozilla-openh264 mpv nautilus-gsconnect neovim network-manager-applet npm nushell
                            power-profiles-daemon rpmfusion-free-release rpmfusion-nonfree-release setzer slurp starship swayidle swaylock thunderbird vlc waybar wireguard-tools
                            wl-clipboard wlogout wmctrl wofi
             LocalPackages: cage-0.1.2.1-4.fc34.x86_64 wlroots-0.13.0-1.fc34.x86_64 sway-1.6-1.fc34.x86_64

  ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210219.n.0 (2021-02-19T08:05:58Z)
                BaseCommit: a75578051e62931d306d256683dd37ed3c54a4b7c76643e25fc53772444d2c81
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: brightnessctl broadcom-wl chromium cronie dconf-editor dialect epiphany ffmpeg file-roller fish gcc gnome-shell-extension-gsconnect gnome-tweaks grim
                            gstreamer1-libav gstreamer1-plugin-openh264 gstreamer1-plugins-bad-freeworld gstreamer1-plugins-base-tools gstreamer1-plugins-good-extras
                            gstreamer1-plugins-ugly gstreamer1-vaapi hunspell-en hunspell-fr hunspell-it kmodtool langpacks-fr lxpolkit mako man-pages-fr mbpfan
                            minecraft-launcher mozilla-openh264 mpv nautilus-gsconnect neovim network-manager-applet rpmfusion-free-release rpmfusion-nonfree-release setzer slurp
                            sway swayidle swaylock vlc waybar wireguard-tools wl-clipboard wlogout wofi
                    Pinned: yes

  ostree://fedora:fedora/33/x86_64/silverblue
                   Version: 33.20210208.0 (2021-02-08T00:58:46Z)
                BaseCommit: 655b20930843360a5945a2711de72d9085b94a05a627d44dd2eb9b273f09a58a
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: brightnessctl broadcom-wl chromium cronie dconf-editor epiphany ffmpeg file-roller fish gcc gnome-tweaks grim gstreamer1-libav
                            gstreamer1-plugin-openh264 gstreamer1-plugins-bad-freeworld gstreamer1-plugins-base-tools gstreamer1-plugins-good-extras gstreamer1-plugins-ugly
                            gstreamer1-vaapi hunspell-en hunspell-fr hunspell-it kmod-wl kmodtool langpacks-fr lxpolkit mako man-pages-fr mbpfan minecraft-launcher
                            mozilla-openh264 mpv neovim network-manager-applet rpmfusion-free-release rpmfusion-nonfree-release setzer slurp sway swayidle swaylock vlc waybar
                            wireguard-tools wl-clipboard wlogout wofi
                    Pinned: yes

AvailableUpdate:
        Version: 34.20210417.n.0 (2021-04-17T08:06:47Z)
         Commit: b42bbb20d10f64faffe7fa411b6b1ba6435476f94810111d7ec5d78f7a0d5f58
   GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           Diff: 9 upgraded, 20 downgraded

Expected vs actual behavior

# rpm-ostree upgrade
...
Writing OSTree commit... done
error: Committing: Writing content object: min-free-space-percent '3%' would be exceeded, at least 74,9 Mo requested

# compsize -x / /var/home/
/var/tmp/flatpak-cache-HB5Y10/org.gnome.Platform-AIOR10: Permission denied
Processed 2809148 files, 1422680 regular extents (2345152 refs), 1627219 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       55%       84G         153G         223G       
none       100%       41G          41G          56G       
zstd        38%       42G         112G         166G       
prealloc   100%       16M          16M          47M

Expected:

# rpm-ostree upgrade
...
Success!

Steps to reproduce it

I'm on Silverblue 34. I compressed my filesystem using btrfs's transparent compression which is awesome. But, this seems to confuse rpm-ostree into thinking there is not enough space to write a commit on the disk. This is problematic because it may force the user into reinstallation since it can't even upgrade rpm-ostree through normal means if the issue is solved (in my case, I could probably clean up older deployments) (Maybe by unlocking the file system and upgrading though rpm? Even then, that may cause some issues with the rpm db?). This is quite severe since the compression is enabled by default on Silverblue 34!

Note: My partition is 249Go and gnome-disks thinks it is 96,9% full.

Would you like to work on the issue?

I am unfortunately unfamiliar with the rpm-ostree code base, though, I am familiar with Rust and some C/C++, so I could eventually help?

jlebon commented 3 years ago

This is more a libostree issue but there's no way to transfer issues between namespaces.

Trying to understand a bit more:

TOTAL 55% 84G 153G 223G

Is this saying that actual disk usage is 55%?

Note: My partition is 249Go and gnome-disks thinks it is 96,9% full.

Hmm, it seems like fstatvfs then isn't returning the true number of free on-disk blocks? Can you also provide the output of stat --file-system / /var/home?

This might be impossible to address fully since there's no way to know ahead of time how compressible the data we're writing will be. But what we could probably do is detect btrfs and use whatever e.g. ioctl calls compsize uses to accurately get free space.

LyesSaadi commented 3 years ago

This is more a libostree issue but there's no way to transfer issues between namespaces.

Yeah, sorry for that. I realized afterward that this was actually an ostree warning, and not a rpm-ostree one. Do I need to file an issue there as well ?

Trying to understand a bit more:

TOTAL 55% 84G 153G 223G

Is this saying that actual disk usage is 55%?

Compsize reports 4 values :

55% here is just saying that my Disk Usage is 55% of what it would have been if it wasn't compressed. Great, isn't !

So, this is actually a problem since f33, but no one noticed before apparently. But the compression made even more unreliable, until I noticed myself that my disk wasn't as full as was reported by gnome-disks and that the numbers ostree was getting were wrong.

Hmm, it seems like fstatvfs then isn't returning the true number of free on-disk blocks? Can you also provide the output of stat --file-system / /var/home?

/ and /var/home are on the same partition, but on different subvolumes.

➜ stat --file-system / /var/home
  Fichier : « / »
 Identif. : c661c77a1bac4f28 Longueur du nom : 255     Type : btrfs
Taille de bloc : 4096       Taille de bloc fondamentale : 4096
 Blocs : total : 60705536   libre : 2159929    disponible : 2009397
Inœuds : total : 0          libre : 0
  Fichier : « /var/home »
 Identif. : c661c77a1bac4f29 Longueur du nom : 255     Type : btrfs
Taille de bloc : 4096       Taille de bloc fondamentale : 4096
 Blocs : total : 60705536   libre : 2159929    disponible : 2009397
Inœuds : total : 0          libre : 0

fstatvfs does indeed seem to get it wrong...

The output is in French, sorry for that... You seem to be in Toronto, so I hope you do remember your French classes :P !

This might be impossible to address fully since there's no way to know ahead of time how compressible the data we're writing will be. But what we could probably do is detect btrfs and use whatever e.g. ioctl calls compsize uses to accurately get free space.

That's also what I thought you could do. It's the only way I could think of to solve this issue permanently...

jlebon commented 3 years ago

fstatvfs does indeed seem to get it wrong...

Hmm indeed, thanks for the output. I wonder if rpm on traditional systems also hits this then (it also does a similar space check).

You can work around this for now by explicitly setting min-free-space-percent to 0 in /ostree/repo/config.

LyesSaadi commented 3 years ago

You can work around this for now by explicitly setting min-free-space-percent to 0 in /ostree/repo/config.

Thank you for this :D! At least there's an easy way to fix it if someone is stuck! I had already removed my local snapshots and stored them on another partition (gnome-disk is happier :P!).

I wonder if rpm on traditional systems also hits this then (it also does a similar space check).

That would be quite dangerous in that case! It's a bit late, but, if this turns out to be true, Fedora could be pushing a broken feature... I haven't found any issue on their GitHub and on the Bugzilla, but I'll send a mail to devel later on... Just to confirm...

LyesSaadi commented 3 years ago

I haven't found any issue on their GitHub and on the Bugzilla, but I'll send a mail to devel later on... Just to confirm...

Sent. Let's hope we are wrong...

LyesSaadi commented 3 years ago

It was an error on my end: compsize wasn't doing the checks recursively and wasn't taking my snapshots into consideration! And because I compressed my files after doing my snapshots, it just duplicated all my files! My filesystem was actually full!