shirou / gopsutil

psutil for golang
Other
10.63k stars 1.59k forks source link

Disk usage Ceph kernel mount not fully working #1344

Closed itsties closed 2 days ago

itsties commented 2 years ago

Describe the bug The fields inodesUsed, inodesUsedPercent, used and usedPercent fields are always 0. The fields total and free are working fine. When I run df -H values are working fine. Only tested on Debian machine.

To Reproduce Mount

# cat /etc/fstab | grep ceph
10.3.10.1,10.3.10.2,10.3.10.3:/volumes/_nogroup/dbe31a78-4059-4df6-a6cc-21a25207768b /mnt/ceph ceph name=ceph-test,secretfile=/etc/ceph/ceph.key,_netdev,noatime 0 0

Test script

// cat test.go
package main
import (
        "fmt"
        "github.com/shirou/gopsutil/v3/disk"
)

func main() {
        v, _ := disk.Usage("/mnt/ceph")
        fmt.Println(v)
}

Output

{
  "path": "/mnt/ceph",
  "fstype": "ceph",
  "total": 107374182400,
  "free": 75061264384,
  "used": 0,
  "usedPercent": 0,
  "inodesTotal": 2167140,
  "inodesUsed": 0,
  "inodesFree": 18446744073709552000,
  "inodesUsedPercent": 0
}

Expected behavior

{
  "path": "/mnt/ceph",
  "fstype": "ceph",
  "total": 107374182400,
  "free": 75061264384,
  "used": 31555584,
  "usedPercent": 31.911490484942368,
  "inodesTotal": 2167140,
  "inodesUsed": 0,
  "inodesFree": 0,
  "inodesUsedPercent": 0
}

Environment (please complete the following information):

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# uname -a
Linux vps2071 5.10.0-17-cloud-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13) x86_64 GNU/Linux

Additional context N/A

shirou commented 2 years ago

disk usage on linux uses golang.org/x/sys/unix.Statfs.

It seems Statfs on ceph has something wrong. Could you this code?

package main

import (
   "fmt"
   "golang.org/x/sys/unix"
)
func main(){
  stat := unix.Statfs_t{}
  unix.Statfs("/mnt/ceph", &stat)
  fmt.Printf("%#v\n", stat)
}
itsties commented 2 years ago

This is my output when running your code

unix.Statfs_t{Type:12805120, Bsize:4194304, Blocks:0x6400, Bfree:0x45e8, Bavail:0x45e8, Files:0x211170, Ffree:0xffffffffffffffff, Fsid:unix.Fsid{Val:[2]int32{-1469057901, -1}}, Namelen:255, Frsize:4194304, Flags:1056, Spare:[4]int64{0, 0, 0, 0}}

or more readable fmt.Printf("%+v\n", stat)

{Type:12805120 Bsize:4194304 Blocks:25600 Bfree:17896 Bavail:17896 Files:2167163 Ffree:18446744073709551615 Fsid:{Val:[-1469057901 -1]} Namelen:255 Frsize:4194304 Flags:1056 Spare:[0 0 0 0]}
itsties commented 2 years ago

I believe this is where is goes wrong with my Files:2167163 & Ffree:18446744073709551615: https://github.com/shirou/gopsutil/blob/ed37dc27a286a25cbe76adf405176c69191a1f37/disk/disk_unix.go#L21-L33

nf-brentsaner commented 3 months ago

(Still?) experiencing this as well on latest release. For now I just subtract UsageStat.Free from UsageStat.Total since both of those fields are populated and correct.

itsties commented 3 months ago

I initially got this issue with using telegraf, but since this fix the fields used and usedPercent are working for me

Can you run this?

package main

import (
   "fmt"
   "golang.org/x/sys/unix"
)
func main(){
  stat := unix.Statfs_t{}
  unix.Statfs("/mnt/ceph", &stat)
  fmt.Printf("%#v\n", stat)
}
nf-brentsaner commented 2 days ago

I initially got this issue with using telegraf, but since this fix the fields used and usedPercent are working for me

Can you run this? (...)

So sorry! Just saw this now.

Statfs_t seems to return correctly for RBD.

For CephFS, though:

unix.Statfs_t{Type:12805120, Bsize:4194304, Blocks:0xa00000, Bfree:0x31b5ac, Bavail:0x31b5ac, Files:0x70c07e, Ffree:0xffffffffffffffff, Fsid:unix.Fsid{Val:[2]int32{-1125264306, 4}}, Namelen:255, Frsize:4194304, Flags:1056, Spare:[4]int64{0, 0, 0, 0}}

I presume the Ffree field and/or Blocks field are what's screwing the pooch here, everything else looks reasonably correct at a glance.

nf-brentsaner commented 2 days ago

Actually-

The bytes usage looks correct now. I might have imported an older/cached version of the module on that program... whoops.

The inode usage is still 0, but that obviously would be the case for CephFS -- it doesn't track inode usage like RBD and real disks do:

[07:56:25] root@[RECACTED]:~ # df -i /mnt/ceph                                                                                                                                                           
Filesystem                                     Inodes IUsed IFree IUse% Mounted on
[REDACTED]:6789,[REDACTED]:6789:/path/subpath 7430326     -     -     - /mnt/ceph

If anything, this is a kernel bug or intentional kernel behavior. I don't have one handy at the moment, but I'm immensely curious to see if the same occurs on NFS and SMB mounts.

nf-brentsaner commented 2 days ago

(SIGH)

Okay, yeah. I was using 3.x, not 4.x. Working for me now. 🙃