Default refreservation overflows for zvol >64TB, volblocksize=16k [libzfs_dataset.c#L5569]

[TestedandBested]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	Server 22.04.3 LTS
Kernel Version	5.15.0-100-generic
Architecture	x64
OpenZFS Version	zfs-2.1.5-1ubuntu6~22.04.2

Describe the problem you're observing

The default refreservation allocation exhibits an overflow for zvols larger than 64TB for volblocksize=16k (default):

NAME           VOLSIZE  REFRESERV
test/deleteme      62T      91.2T
test/deleteme      63T      92.6T
test/deleteme      64T      65.0T
test/deleteme      65T      2.47T
test/deleteme      66T      3.94T

This allows volumes to be created with insufficient available space in the zpool. I'm raising this on behalf of the person who investigated it (thanks to reddit.com/user/Dagger0)

Describe how to reproduce the problem

Create a zvol larger than 64TB with default volblocksize 16k. This was tested on a 12-wide RAIDZ3 zpool of 10TB disks, zpool created with ashift=12.

This example shows increments in the size of the zvol by 5TB each time with an extra examples around 64TB.
I get the correct 'out of space' error when there is insufficient space in the zpool until I create a volume larger than 64TB at which time the refreservation value overflows. All examples from 64.1TB onwards exhibit this behaviour.

nas-01:~# zfs list
NAME             USED  AVAIL     REFER  MOUNTPOINT
ZFSNAS01_zpool  1.77M  79.2T      279K  /ZFSNAS01_zpool

nas-01:~# zfs create -V 45T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol;
ZFSNAS01_zpool/Testvol  refreservation  66.2T      local
nas-01:~# zfs create -V 50T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol;
ZFSNAS01_zpool/Testvol  refreservation  73.5T      local
nas-01:~# zfs create -V 55T  ZFSNAS01_zpool/Testvol
cannot create 'ZFSNAS01_zpool/Testvol': out of space
nas-01:~# zfs create -V 60T  ZFSNAS01_zpool/Testvol
cannot create 'ZFSNAS01_zpool/Testvol': out of space
nas-01:~# zfs create -V 63.9T  ZFSNAS01_zpool/Testvol
cannot create 'ZFSNAS01_zpool/Testvol': out of space
nas-01:~# zfs create -V 64T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol
ZFSNAS01_zpool/Testvol  refreservation  65.0T      local
nas-01:~# zfs create -V 64.1T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol
ZFSNAS01_zpool/Testvol  refreservation  1.15T      local
nas-01:~# zfs create -V 64.2T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol
ZFSNAS01_zpool/Testvol  refreservation  1.30T      local
nas-01:~# zfs create -V 64.3T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol
ZFSNAS01_zpool/Testvol  refreservation  1.44T      local
nas-01:~# zfs create -V 64.4T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol
ZFSNAS01_zpool/Testvol  refreservation  1.59T      local
nas-01:~# zfs create -V 65T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol;
ZFSNAS01_zpool/Testvol  refreservation  2.47T      local
nas-01:~# zfs create -V 70T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol;
ZFSNAS01_zpool/Testvol  refreservation  9.82T      local
nas-01:~# zfs create -V 75T  ZFSNAS01_zpool/Testvol; zfs get -H refreservation ZFSNAS01_zpool/Testvol;
ZFSNAS01_zpool/Testvol  refreservation  17.2T      local

Reddit user Dagger0 reported the following:

I tracked it down to this calculation: libzfs_dataset.c#L5569 which overflows uint64_t when nblocks * asize > 128T.

[rincebrain]

Amusingly, I posted the PR fixing this before you opened the bug.

15996

[TestedandBested]

Amusingly, I posted the PR fixing this before you opened the bug. #15996

That's great news, thanks. What is the etiquette at this point, should I close this with a comment referencing #15996?