rincebrain
commented
7 months ago "TRIM"
Open luckylinux opened 7 months ago
rincebrain
commented
7 months ago "TRIM"
luckylinux
commented
7 months ago It should be disabled on the host because it's ZFS on top of LUKS. That's the default behavior from what I understood at least.
However systemctl status fstrim.timer reports on the Host:
● fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; preset: enabled)
Active: active (waiting) since Fri 2024-04-05 10:29:41 CEST; 4 days ago
Trigger: Mon 2024-04-15 01:37:09 CEST; 5 days left
Triggers: ● fstrim.service
Docs: man:fstrim
Apr 05 10:29:41 pve16 systemd[1]: Started fstrim.timer - Discard unused blocks once a week.On the guest you might be right though
I always have SSD Emulation + Discard + IO thread enabled on all of my VMs.
But zpool get autotrim returns off both for the Host and the Guest VM.
systemctl status fstrim.timer reports on the Guest VM:
○ fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/lib/systemd/system/fstrim.timer; disabled; preset: enabled)
Active: inactive (dead)
Trigger: n/a
Triggers: ● fstrim.service
Docs: man:fstrimAny other command to check ?
rincebrain
commented
7 months ago You misunderstood.
fstrim doesn't do anything with ZFS, and absent autotrim, it's not going to issue such requests without an explicit zpool trim in the guest, leaving the space that was freed in the guest still marked in use on the host.
luckylinux
commented
7 months ago And that's ZFS-specific ? I mean the ext4 partition for / on top the the ZVOL (like many other containers I have) do not really have this problem.
Somewhere I think I read that zpool trim is kinda dangerous concerning data loss. Isn't it ?
rincebrain
commented
7 months ago Yes, the command zpool trim is ZFS specific.
There was an uncommon race with data mangling using any kind of TRIM that was fixed in 2.2 and 2.1.14.
I wouldn't suggest using any FS that you don't want to use TRIM with inside a VM if you're worried about the space usage when things are freed and not deleted on the host.
luckylinux
commented
7 months ago What do you mean exactly by your latest statement ? That I should run zpool trim or that I should not be running ZFS on top of ZVOL ?
rincebrain
commented
7 months ago luckylinux
commented
7 months ago Thanks.
Hopefully there won't be any regression of that bug :D.
However nothing seems to be happening.
Issued zpool trim zdata on the Guest and zpool status -t reports on the Guest:
pool: zdata
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:07 with 0 errors on Sun Mar 10 00:24:09 2024
config:
NAME STATE READ WRITE CKSUM
zdata ONLINE 0 0 0
PODMAN ONLINE 0 0 0 (100% trimmed, completed at Wed 10 Apr 2024 12:52:57 AM CEST)
errors: No known data errors
Host zfs list | grep "vm-103-disk-1" on the Host:
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-103-disk-1 67.8G 274G 989M -Granted it could be running in the background. But for now there is absolutely no change.
rincebrain
commented
7 months ago You may note that there are 3 columns there, and "referenced" changed pretty substantially.
luckylinux
commented
7 months ago True. So I just need to destroy the old snapshots of that dataset on the Host.
luckylinux
commented
7 months ago Yep.
Now zfs list | grep "vm-103-disk-1" yields:
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-103-disk-1 989M 312G 989M -Maybe the root filesystem of the Guest VM (ext4) has autotrim enabled by default then ?
That could explain the behavior ...
rincebrain
commented
7 months ago For reference, you could have seen if that was going to happen before doing it with zfs destroy -nv [list of snapshots] or by looking at the 4 different "usedby" properties which sum to USED.
zpool trim is for ZFS. The root FS is ext4, so fstrim will do as you expect.
luckylinux
commented
7 months ago Or if it's safe enough enable ZFS autotrim on the Guest VM ?
zpool set autotrim=on zdataThe bug wasn't specific to automatic or manual trim, afair, so autotrim should be no less safe than manual trim.
kwinz
commented
7 months ago There was an uncommon race with data mangling using any kind of TRIM that was fixed in 2.2 and 2.1.14.
There seems to be a new issue in 2.2.3: #16056
Just so you're aware
rincebrain
commented
7 months ago Kind of.
Note that #16056, from my quick reading, seems to be the result of using hardware that lied and gave an invalid value for how big TRIM can be, combined with a failure in error case handling for that since that should not really ever happen. #16070 fixes the latter, but it's not entirely clear what the right thing to do if I'm right about the former is.
System information
Host (Proxmox VE):
Guest VM (Debian GNU/Linux KVM):
Describe the problem you're observing
I am facing some EXTREME overhead when creating a ZFS zpool in a Proxmox VE VM Guest on top of a ZVOL on the Host.
The ZFS Pool on the Host System is also sitting on top of a LUKS / Cryptsetup Full-Disk-Encryption, although I do NOT think this is relevant for the issue described here (since the Host ZFS Pool is sitting on top of the DMCrypt / LUKS Device).
zfs liston the host:zpool get all rpoolHost Pool Properties:zfs get all rpool/data/vm-103-disk-1Host ZFS Properties:vm-103-disk-0is ext4 on top of ZVOL for comparison purposes.vm-103-disk-1is the ZFS Pool on top of ZVOL which has the issue.df -ahfor / Guest VM filesystemvm-103-disk-0(for comparison purposes) which is based on:So 5.7G used on the Guest and 8.27G on the Host. Overhead is roughly 45% (8.27G/5.7G - 1)*100%.
zfs liston the Guest VMvm-103-disk-1Container Data Storage (Podman):Overhead is roughly 8350% (67.6G/0.8G - 1)*100% !!!
zpool get all zdataGuest VM Pool Properties:zfs get all zdataGuest ZFS Properties:Note: it is possible that this issue is caused by block size / volblocksize or similar parameter, since Podman / Docker containers could generate lots of small files.
That does not sound like much though ... On the GUEST recordsize is set to 128K, that's probably a bit high, isn't it ?
Regardless, just because of the number of files, that would yield: 14475 x 128K = 1852800K = 1852.8 M = 1.85 G
So probably it's causing some overhead inside the guest, but not on the level of the overhead between guest and host ...
Describe how to reproduce the problem
I don't think that it's necessary to have a VM to replicate this.
Probably just on one system (Host) it's sufficient to create a ZFS Pool on top of the ZVOL.
I disabled compression on the Guest level since it would only cause additional CPU load for no apparent benefit. Therefore compression should NOT be the cause of this huge overhead.
Notes
The idea of having a ZFS Pool on top of the ZVOL is have better control over ZFS Snapshots. In this case, after many other things will have been configured correctly, the snapshot & backup plan of
rpool/data/vm-103-disk-1could be performed by the Guest, as opposed to the host for many other VMs.This can avoid backing up non-useful Data (such as Container Images or Container Storage) and only backup Useful / Critical Data (Containers Configuration, Secrets, Data, Certificates, Volumes, ...) thus saving a lot on Disk Space on the Backup Server.
Include any warning/errors/backtraces from the system logs