Disk usage Ceph kernel mount not fully working

[itsties]

Describe the bug The fields inodesUsed, inodesUsedPercent, used and usedPercent fields are always 0. The fields total and free are working fine. When I run df -H values are working fine. Only tested on Debian machine.

To Reproduce Mount

# cat /etc/fstab | grep ceph
10.3.10.1,10.3.10.2,10.3.10.3:/volumes/_nogroup/dbe31a78-4059-4df6-a6cc-21a25207768b /mnt/ceph ceph name=ceph-test,secretfile=/etc/ceph/ceph.key,_netdev,noatime 0 0

Test script

// cat test.go
package main
import (
        "fmt"
        "github.com/shirou/gopsutil/v3/disk"
)

func main() {
        v, _ := disk.Usage("/mnt/ceph")
        fmt.Println(v)
}

Output

{
  "path": "/mnt/ceph",
  "fstype": "ceph",
  "total": 107374182400,
  "free": 75061264384,
  "used": 0,
  "usedPercent": 0,
  "inodesTotal": 2167140,
  "inodesUsed": 0,
  "inodesFree": 18446744073709552000,
  "inodesUsedPercent": 0
}

Expected behavior

{
  "path": "/mnt/ceph",
  "fstype": "ceph",
  "total": 107374182400,
  "free": 75061264384,
  "used": 31555584,
  "usedPercent": 31.911490484942368,
  "inodesTotal": 2167140,
  "inodesUsed": 0,
  "inodesFree": 0,
  "inodesUsedPercent": 0
}

Environment (please complete the following information):

• [x] Linux: [paste contents of /etc/os-release and the result of uname -a]

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# uname -a
Linux vps2071 5.10.0-17-cloud-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13) x86_64 GNU/Linux

Additional context N/A

[shirou]

disk usage on linux uses golang.org/x/sys/unix.Statfs.

It seems Statfs on ceph has something wrong. Could you this code?

package main

import (
   "fmt"
   "golang.org/x/sys/unix"
)
func main(){
  stat := unix.Statfs_t{}
  unix.Statfs("/mnt/ceph", &stat)
  fmt.Printf("%#v\n", stat)
}

[itsties]

This is my output when running your code

unix.Statfs_t{Type:12805120, Bsize:4194304, Blocks:0x6400, Bfree:0x45e8, Bavail:0x45e8, Files:0x211170, Ffree:0xffffffffffffffff, Fsid:unix.Fsid{Val:[2]int32{-1469057901, -1}}, Namelen:255, Frsize:4194304, Flags:1056, Spare:[4]int64{0, 0, 0, 0}}

or more readable fmt.Printf("%+v\n", stat)

{Type:12805120 Bsize:4194304 Blocks:25600 Bfree:17896 Bavail:17896 Files:2167163 Ffree:18446744073709551615 Fsid:{Val:[-1469057901 -1]} Namelen:255 Frsize:4194304 Flags:1056 Spare:[0 0 0 0]}

[itsties]

I believe this is where is goes wrong with my Files:2167163 & Ffree:18446744073709551615: https://github.com/shirou/gopsutil/blob/ed37dc27a286a25cbe76adf405176c69191a1f37/disk/disk_unix.go#L21-L33

[nf-brentsaner]

(Still?) experiencing this as well on latest release. For now I just subtract UsageStat.Free from UsageStat.Total since both of those fields are populated and correct.

[itsties]

I initially got this issue with using telegraf, but since this fix the fields used and usedPercent are working for me

Can you run this?

package main

import (
   "fmt"
   "golang.org/x/sys/unix"
)
func main(){
  stat := unix.Statfs_t{}
  unix.Statfs("/mnt/ceph", &stat)
  fmt.Printf("%#v\n", stat)
}

[nf-brentsaner]

I initially got this issue with using telegraf, but since this fix the fields used and usedPercent are working for me

Can you run this? (...)

So sorry! Just saw this now.

Statfs_t seems to return correctly for RBD.

For CephFS, though:

unix.Statfs_t{Type:12805120, Bsize:4194304, Blocks:0xa00000, Bfree:0x31b5ac, Bavail:0x31b5ac, Files:0x70c07e, Ffree:0xffffffffffffffff, Fsid:unix.Fsid{Val:[2]int32{-1125264306, 4}}, Namelen:255, Frsize:4194304, Flags:1056, Spare:[4]int64{0, 0, 0, 0}}

I presume the Ffree field and/or Blocks field are what's screwing the pooch here, everything else looks reasonably correct at a glance.

[nf-brentsaner]

Actually-

The bytes usage looks correct now. I might have imported an older/cached version of the module on that program... whoops.

The inode usage is still 0, but that obviously would be the case for CephFS -- it doesn't track inode usage like RBD and real disks do:

[07:56:25] root@[RECACTED]:~ # df -i /mnt/ceph                                                                                                                                                           
Filesystem                                     Inodes IUsed IFree IUse% Mounted on
[REDACTED]:6789,[REDACTED]:6789:/path/subpath 7430326     -     -     - /mnt/ceph

If anything, this is a kernel bug or intentional kernel behavior. I don't have one handy at the moment, but I'm immensely curious to see if the same occurs on NFS and SMB mounts.

[nf-brentsaner]

(SIGH)

Okay, yeah. I was using 3.x, not 4.x. Working for me now. 🙃