openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.57k stars 1.74k forks source link

ZFS is unable to release space properly #16043

Open ChaseTheDark opened 7 months ago

ChaseTheDark commented 7 months ago

System information

Type Version/Name
Distribution Name Proxmox ( Debian )
Distribution Version 8.1.5 ( 12 )
Kernel Version 6.5.11-8-pve
Architecture x86_64
OpenZFS Version 2.2.2

Describe the problem you're observing

After deleting a snapshot, I noticed that my space wasn't released and there were no signs of it being released.

Initially, I loaded a lot of data into zpool/PVE and then started setting up my ZFS. Here are the commands I used initially.

zfs snapshot zpool@firstSnapshot
zfs list -t snapshot
zfs destroy zpool@firstSnapshot
zfs list -t snapshot 
zfs create -p zpool/PVE/multimedia/videos
zfs destroy -r zpool/PVE
zfs create -p zpool/PVE/multimedia/videos
zfs set compression=off zpool/PVE/multimedia/videos
zfs get -r compression zpool
zfs destroy -r zpool/PVE
zfs list
zfs list -t snapshot 
zfs snapshot zpool@firstSnapshot
zfs list -t snapshot 
zfs create -p zpool/PVE/multimedia/videos
zfs set compression=off zpool/PVE/multimedia/videos
zfs list
zfs get -r compression zpool
zfs list -t snapshot 

I am a complete ZFS novice, and after running the above commands, I found that I lost my data, and attempts to roll back were ineffective.zfs rollback -r zpool@firstSnapshot

Luckily, I eventually found my files in a hidden folder.Before retrieving my files, I used the command zfs list to view my datasets. There were two: zpool and zpool/PVE.

cd /zpool/.zfs/snapshot/firstSnapshot
cp -r ./PVE /zpool
zfs snapshot zpool/PVE@PVE-2024.03.29
zfs list -t snapshot
zfs destroy zpool@firstSnapshot

Normally, space should be released after deleting the snapshot zpool@firstSnapshot, but it didn't happen. Below is some information I checked. Note: There are only about 100MB of files in the directory 'zpool/'.


zfs list -t all -o space

$ zfs list -t all -o space
NAME                      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zpool                     11.4T  2.99T        0B   1.49T             0B      1.50T
zpool/PVE                 11.4T  1.50T      202K   1.50T             0B         0B
zpool/PVE@PVE-2024.03.29      -   202K         -       -              -          -

du -sh /zpool

$ du -sh /zpool
1.5T    /zpool

zfs get all

$ zfs get all
NAME                      PROPERTY              VALUE                     SOURCE
zpool                     type                  filesystem                -
zpool                     creation              Tue Mar  5 19:36 2024     -
zpool                     used                  2.99T                     -
zpool                     available             11.4T                     -
zpool                     referenced            1.49T                     -
zpool                     compressratio         1.03x                     -
zpool                     mounted               yes                       -
zpool                     quota                 none                      default
zpool                     reservation           none                      default
zpool                     recordsize            128K                      default
zpool                     mountpoint            /zpool                    default
zpool                     sharenfs              off                       default
zpool                     checksum              on                        default
zpool                     compression           lz4                       local
zpool                     atime                 off                       local
zpool                     devices               on                        default
zpool                     exec                  on                        default
zpool                     setuid                on                        default
zpool                     readonly              off                       default
zpool                     zoned                 off                       default
zpool                     snapdir               hidden                    default
zpool                     aclmode               discard                   default
zpool                     aclinherit            restricted                default
zpool                     createtxg             1                         -
zpool                     canmount              on                        default
zpool                     xattr                 on                        default
zpool                     copies                1                         default
zpool                     version               5                         -
zpool                     utf8only              off                       -
zpool                     normalization         none                      -
zpool                     casesensitivity       sensitive                 -
zpool                     vscan                 off                       default
zpool                     nbmand                off                       default
zpool                     sharesmb              off                       default
zpool                     refquota              none                      default
zpool                     refreservation        none                      default
zpool                     guid                  17902915914539698946      -
zpool                     primarycache          all                       default
zpool                     secondarycache        all                       default
zpool                     usedbysnapshots       0B                        -
zpool                     usedbydataset         1.49T                     -
zpool                     usedbychildren        1.50T                     -
zpool                     usedbyrefreservation  0B                        -
zpool                     logbias               latency                   default
zpool                     objsetid              54                        -
zpool                     dedup                 off                       default
zpool                     mlslabel              none                      default
zpool                     sync                  standard                  default
zpool                     dnodesize             legacy                    default
zpool                     refcompressratio      1.03x                     -
zpool                     written               1.49T                     -
zpool                     logicalused           3.08T                     -
zpool                     logicalreferenced     1.53T                     -
zpool                     volmode               default                   default
zpool                     filesystem_limit      none                      default
zpool                     snapshot_limit        none                      default
zpool                     filesystem_count      none                      default
zpool                     snapshot_count        none                      default
zpool                     snapdev               hidden                    default
zpool                     acltype               off                       default
zpool                     context               none                      default
zpool                     fscontext             none                      default
zpool                     defcontext            none                      default
zpool                     rootcontext           none                      default
zpool                     relatime              on                        default
zpool                     redundant_metadata    all                       default
zpool                     overlay               on                        default
zpool                     encryption            off                       default
zpool                     keylocation           none                      default
zpool                     keyformat             none                      default
zpool                     pbkdf2iters           0                         default
zpool                     special_small_blocks  0                         default
zpool                     snapshots_changed     Sat Mar 30  0:24:15 2024  -
zpool/PVE                 type                  filesystem                -
zpool/PVE                 creation              Thu Mar 28  4:51 2024     -
zpool/PVE                 used                  1.50T                     -
zpool/PVE                 available             11.4T                     -
zpool/PVE                 referenced            1.50T                     -
zpool/PVE                 compressratio         1.03x                     -
zpool/PVE                 mounted               yes                       -
zpool/PVE                 quota                 none                      default
zpool/PVE                 reservation           none                      default
zpool/PVE                 recordsize            128K                      default
zpool/PVE                 mountpoint            /zpool/PVE                default
zpool/PVE                 sharenfs              off                       default
zpool/PVE                 checksum              on                        default
zpool/PVE                 compression           lz4                       inherited from zpool
zpool/PVE                 atime                 off                       inherited from zpool
zpool/PVE                 devices               on                        default
zpool/PVE                 exec                  on                        default
zpool/PVE                 setuid                on                        default
zpool/PVE                 readonly              off                       default
zpool/PVE                 zoned                 off                       default
zpool/PVE                 snapdir               hidden                    default
zpool/PVE                 aclmode               discard                   default
zpool/PVE                 aclinherit            restricted                default
zpool/PVE                 createtxg             62674                     -
zpool/PVE                 canmount              on                        default
zpool/PVE                 xattr                 on                        default
zpool/PVE                 copies                1                         default
zpool/PVE                 version               5                         -
zpool/PVE                 utf8only              off                       -
zpool/PVE                 normalization         none                      -
zpool/PVE                 casesensitivity       sensitive                 -
zpool/PVE                 vscan                 off                       default
zpool/PVE                 nbmand                off                       default
zpool/PVE                 sharesmb              off                       default
zpool/PVE                 refquota              none                      default
zpool/PVE                 refreservation        none                      default
zpool/PVE                 guid                  6699107644573188210       -
zpool/PVE                 primarycache          all                       default
zpool/PVE                 secondarycache        all                       default
zpool/PVE                 usedbysnapshots       202K                      -
zpool/PVE                 usedbydataset         1.50T                     -
zpool/PVE                 usedbychildren        0B                        -
zpool/PVE                 usedbyrefreservation  0B                        -
zpool/PVE                 logbias               latency                   default
zpool/PVE                 objsetid              611                       -
zpool/PVE                 dedup                 off                       default
zpool/PVE                 mlslabel              none                      default
zpool/PVE                 sync                  standard                  default
zpool/PVE                 dnodesize             legacy                    default
zpool/PVE                 refcompressratio      1.03x                     -
zpool/PVE                 written               10.9G                     -
zpool/PVE                 logicalused           1.55T                     -
zpool/PVE                 logicalreferenced     1.55T                     -
zpool/PVE                 volmode               default                   default
zpool/PVE                 filesystem_limit      none                      default
zpool/PVE                 snapshot_limit        none                      default
zpool/PVE                 filesystem_count      none                      default
zpool/PVE                 snapshot_count        none                      default
zpool/PVE                 snapdev               hidden                    default
zpool/PVE                 acltype               off                       default
zpool/PVE                 context               none                      default
zpool/PVE                 fscontext             none                      default
zpool/PVE                 defcontext            none                      default
zpool/PVE                 rootcontext           none                      default
zpool/PVE                 relatime              on                        default
zpool/PVE                 redundant_metadata    all                       default
zpool/PVE                 overlay               on                        default
zpool/PVE                 encryption            off                       default
zpool/PVE                 keylocation           none                      default
zpool/PVE                 keyformat             none                      default
zpool/PVE                 pbkdf2iters           0                         default
zpool/PVE                 special_small_blocks  0                         default
zpool/PVE                 snapshots_changed     Fri Mar 29 21:23:34 2024  -
zpool/PVE@PVE-2024.03.29  type                  snapshot                  -
zpool/PVE@PVE-2024.03.29  creation              Fri Mar 29 21:23 2024     -
zpool/PVE@PVE-2024.03.29  used                  202K                      -
zpool/PVE@PVE-2024.03.29  referenced            1.49T                     -
zpool/PVE@PVE-2024.03.29  compressratio         1.03x                     -
zpool/PVE@PVE-2024.03.29  devices               on                        default
zpool/PVE@PVE-2024.03.29  exec                  on                        default
zpool/PVE@PVE-2024.03.29  setuid                on                        default
zpool/PVE@PVE-2024.03.29  createtxg             70252                     -
zpool/PVE@PVE-2024.03.29  xattr                 on                        default
zpool/PVE@PVE-2024.03.29  version               5                         -
zpool/PVE@PVE-2024.03.29  utf8only              off                       -
zpool/PVE@PVE-2024.03.29  normalization         none                      -
zpool/PVE@PVE-2024.03.29  casesensitivity       sensitive                 -
zpool/PVE@PVE-2024.03.29  nbmand                off                       default
zpool/PVE@PVE-2024.03.29  guid                  13386084269841811841      -
zpool/PVE@PVE-2024.03.29  primarycache          all                       default
zpool/PVE@PVE-2024.03.29  secondarycache        all                       default
zpool/PVE@PVE-2024.03.29  defer_destroy         off                       -
zpool/PVE@PVE-2024.03.29  userrefs              0                         -
zpool/PVE@PVE-2024.03.29  objsetid              478                       -
zpool/PVE@PVE-2024.03.29  mlslabel              none                      default
zpool/PVE@PVE-2024.03.29  refcompressratio      1.03x                     -
zpool/PVE@PVE-2024.03.29  written               1.49T                     -
zpool/PVE@PVE-2024.03.29  logicalreferenced     1.53T                     -
zpool/PVE@PVE-2024.03.29  acltype               off                       default
zpool/PVE@PVE-2024.03.29  context               none                      default
zpool/PVE@PVE-2024.03.29  fscontext             none                      default
zpool/PVE@PVE-2024.03.29  defcontext            none                      default
zpool/PVE@PVE-2024.03.29  rootcontext           none                      default
zpool/PVE@PVE-2024.03.29  encryption            off                       default

zpool get all

$ zpool get all
NAME   PROPERTY                       VALUE                          SOURCE
zpool  size                           21.8T                          -
zpool  capacity                       20%                            -
zpool  altroot                        -                              default
zpool  health                         ONLINE                         -
zpool  guid                           17372564618608095265           -
zpool  version                        -                              default
zpool  bootfs                         -                              default
zpool  delegation                     on                             default
zpool  autoreplace                    off                            default
zpool  cachefile                      -                              default
zpool  failmode                       wait                           default
zpool  listsnapshots                  off                            default
zpool  autoexpand                     off                            default
zpool  dedupratio                     1.00x                          -
zpool  free                           17.3T                          -
zpool  allocated                      4.48T                          -
zpool  readonly                       off                            -
zpool  ashift                         12                             local
zpool  comment                        -                              default
zpool  expandsize                     -                              -
zpool  freeing                        0                              -
zpool  fragmentation                  0%                             -
zpool  leaked                         0                              -
zpool  multihost                      off                            default
zpool  checkpoint                     -                              -
zpool  load_guid                      6037510390726762895            -
zpool  autotrim                       off                            default
zpool  compatibility                  off                            default
zpool  bcloneused                     0                              -
zpool  bclonesaved                    0                              -
zpool  bcloneratio                    1.00x                          -
zpool  feature@async_destroy          enabled                        local
zpool  feature@empty_bpobj            active                         local
zpool  feature@lz4_compress           active                         local
zpool  feature@multi_vdev_crash_dump  enabled                        local
zpool  feature@spacemap_histogram     active                         local
zpool  feature@enabled_txg            active                         local
zpool  feature@hole_birth             active                         local
zpool  feature@extensible_dataset     active                         local
zpool  feature@embedded_data          active                         local
zpool  feature@bookmarks              enabled                        local
zpool  feature@filesystem_limits      enabled                        local
zpool  feature@large_blocks           enabled                        local
zpool  feature@large_dnode            enabled                        local
zpool  feature@sha512                 enabled                        local
zpool  feature@skein                  enabled                        local
zpool  feature@edonr                  enabled                        local
zpool  feature@userobj_accounting     active                         local
zpool  feature@encryption             enabled                        local
zpool  feature@project_quota          active                         local
zpool  feature@device_removal         enabled                        local
zpool  feature@obsolete_counts        enabled                        local
zpool  feature@zpool_checkpoint       enabled                        local
zpool  feature@spacemap_v2            active                         local
zpool  feature@allocation_classes     enabled                        local
zpool  feature@resilver_defer         enabled                        local
zpool  feature@bookmark_v2            enabled                        local
zpool  feature@redaction_bookmarks    enabled                        local
zpool  feature@redacted_datasets      enabled                        local
zpool  feature@bookmark_written       enabled                        local
zpool  feature@log_spacemap           active                         local
zpool  feature@livelist               enabled                        local
zpool  feature@device_rebuild         enabled                        local
zpool  feature@zstd_compress          enabled                        local
zpool  feature@draid                  enabled                        local
zpool  feature@zilsaxattr             active                         local
zpool  feature@head_errlog            active                         local
zpool  feature@blake3                 enabled                        local
zpool  feature@block_cloning          enabled                        local
zpool  feature@vdev_zaps_v2           active                         local

zpool status

$ zpool status
  pool: zpool
 state: ONLINE
  scan: scrub repaired 0B in 02:42:24 with 0 errors on Fri Mar 29 23:50:10 2024
config:

        NAME                              STATE     READ WRITE CKSUM
        zpool                             ONLINE       0     0     0
          raidz1-0                        ONLINE       0     0     0
            ata-HUH728080ALE601_VLKEWENY  ONLINE       0     0     0
            ata-HUH728080ALE601_VJGHMJ0X  ONLINE       0     0     0
            ata-HUH728080ALE601_VLJXGN2Y  ONLINE       0     0     0

errors: No known data errors
rincebrain commented 7 months ago

If I were to guess, you have all the contents of the filesystem zpool/PVE also stored in the folder "PVE" on the filesystem `zpool", not just in the filesystem "zpool/PVE" currently mounted there.

ChaseTheDark commented 7 months ago

If I were to guess, you have all the contents of the filesystem zpool/PVE also stored in the folder "PVE" on the filesystem `zpool", not just in the filesystem "zpool/PVE" currently mounted there.

I agree with your speculation. I also believe that's the case, but currently, I can't access my original 1.49TB data by any means, and the 'du' command shows the space usage correctly.

$ du -sh /zpool/*
129M    /zpool/backup
1.5T    /zpool/PVE
------
$ du -sh /zpool/.zfs
0       /zpool/.zfs

In fact, I think the problem occurred with the commands I used initially:

zfs snapshot zpool@firstSnapshot
zfs destroy zpool@firstSnapshot
......
zfs destroy -r zpool/PVE
......
zfs snapshot zpool@firstSnapshot
......

After initially deleting 'zpool/PVE', my data didn't immediately disappear when I checked at that time. That's why I continued setting other properties instead of immediately restoring files.

Furthermore, something strange happened. After losing my data, I tried using the command 'zfs rollback -r zpool@firstSnapshot'. It didn't return any errors, but it also didn't restore any files. I eventually recovered my data using the 'cp' command after finding 'zpool/.zfs'.I suspected that the space occupation was related to my snapshot, so I deleted the zpool@firstSnapshot.

rincebrain commented 7 months ago

So, if you have data in the folder "PVE" under the dataset zpool/, then you could unmount the dataset zpool/PVE, to access the contents of the place where it was mounted, e.g. zfs umount zpool/PVE

ChaseTheDark commented 7 months ago

So, if you have data in the folder "PVE" under the dataset , then you could unmount the dataset , to access the contents of the place where it was mounted, e.g. zpool/``zpool/PVE``zfs umount zpool/PVE

I see, indeed, there are two mount points. However, even after using "zfs umount zpool/PVE," the space still wasn't released.

Let me clarify my space usage:

zpool: Occupies 2.99TB, with subdataset using 1.5TB, and itself using 1.49TB.

rincebrain commented 7 months ago

I wasn't saying it would be released if you did that, I was saying that it would be visible for you to delete then.

It may become clearer to you what can be done if you change the mountpoint of zpool/PVE to, say, /zpool/PVE_dataset, then look at the space usage with du -sh /zpool/ versus du -shx /zpool/.

du -x can also tell you how much space without crossing into things mounted under it.

ChaseTheDark commented 7 months ago

I wasn't saying it would be released if you did that, I was saying that it would be visible for you to delete then.

It may become clearer to you what can be done if you change the mountpoint of to, say, , then look at the space usage with versus .zpool/PVE/zpool/PVE_datasetdu -sh /zpool/du -shx /zpool/

du -x can also tell you how much space without crossing into things mounted under it./zpool/`du -shx /zpool/

I changed the mount point of ”zpool/PVE“. Here are my subsequent observations:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
......
zpool                  13T  1.5T   12T  12% /zpool
......
zpool/PVE              13T  1.5T   12T  12% /zpool/PVE_dataset

$ du -sh /zpool/
3.0T    /zpool/

$ zfs list -t all -o space
NAME                      AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
zpool                     11.4T  2.99T        0B   1.49T             0B      1.50T
zpool/PVE                 11.4T  1.50T      202K   1.50T             0B         0B
zpool/PVE@PVE-2024.03.29      -   202K         -       -              -          -

$ du -sh /zpool/*
129M    /zpool/backup
1.5T    /zpool/PVE
1.5T    /zpool/PVE_dataset

$ du -shx /zpool/
1.5T    /zpool/
rincebrain commented 7 months ago

Yes.

You can see now how /zpool/PVE says it contains 1.5T of data.

You can delete that, if all of it is in /zpool/PVE_dataset.

ChaseTheDark commented 7 months ago

Yes.

You can see now how /zpool/PVE says it contains 1.5T of data.

You can delete that, if all of it is in /zpool/PVE_dataset.

I want to release the space occupied by the 1.49TB dataset rather than the 1.5TB one. In fact, the extra 0.01TB of data was added by me to distinguish between my current data and the previous data.

rincebrain commented 7 months ago

Yes.

You currently have the copy of the data you care about in /zpool/PVE_dataset, and the copy you do not in /zpool/PVE.

You have an rm command. You can use it.