search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.69k stars 1.76k forks source link

zpool status reporting inconsistent error information #16737

Open stuartthebruce opened 2 weeks ago

stuartthebruce commented 2 weeks ago

System information

Type Version/Name
Distribution Name Rocky Linux
Distribution Version 8.10
Kernel Version 4.18.0-553.22.1.el8_10.x86_64
Architecture x86_64
OpenZFS Version 2.2.6

Describe the problem you're observing

zpool status is reporting inconsistent error information

Describe how to reproduce the problem

After zpool status initially reported errors: 1 data errors, use '-v' for a list a zpool scrub was run. When that completed the same error was reported,

ZFS has finished a scrub:

  eid: 121843
class: scrub_finish
 host: zfs4
 time: 2024-11-08 18:56:33-0800
 pool: home3
state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
  see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
 scan: scrub repaired 0B in 4 days 11:37:41 with 0 errors on Fri Nov  8 18:56:33 2024
config:

    NAME                                 STATE     READ WRITE CKSUM
    home3                                ONLINE       0     0     0
      raidz3-0                           ONLINE       0     0     0
        wwn-0x5000cca2531dd934           ONLINE       0     0     0
        wwn-0x5000cca2531e2a94           ONLINE       0     0     0
        wwn-0x5000cca29187c34c           ONLINE       0     0     0
        wwn-0x5000cca2b8482d4c           ONLINE       0     0     0
        wwn-0x5000cca26fcec890           ONLINE       0     0     0
        wwn-0x5000cca2531e5f6c           ONLINE       0     0     0
        wwn-0x5000cca27003bb00           ONLINE       0     0     0
        wwn-0x5000cca2531e84ec           ONLINE       0     0     0
        wwn-0x5000cca2531e868c           ONLINE       0     0     0
        wwn-0x5000cca2531e87d8           ONLINE       0     0     0
        wwn-0x5000cca2531e9750           ONLINE       0     0     0
        wwn-0x5000cca2b0d97f08           ONLINE       0     0     0
        wwn-0x5000cca2531e9f48           ONLINE       0     0     0
        wwn-0x5000cca2531eb96c           ONLINE       0     0     0
        wwn-0x5000cca2531ec858           ONLINE       0     0     0
      raidz3-1                           ONLINE       0     0     0
        wwn-0x5000cca2530aa110           ONLINE       0     0     0
        wwn-0x5000cca2530aa424           ONLINE       0     0     0
        wwn-0x5000cca2530aacb4           ONLINE       0     0     0
        wwn-0x5000cca2530e297c           ONLINE       0     0     0
        wwn-0x5000cca2530f661c           ONLINE       0     0     0
        wwn-0x5000cca291abaed0           ONLINE       0     0     0
        wwn-0x5000cca253123c08           ONLINE       0     0     0
        wwn-0x5000cca253158878           ONLINE       0     0     0
        wwn-0x5000cca253168af4           ONLINE       0     0     0
        wwn-0x5000cca2b0044388           ONLINE       0     0     0
        wwn-0x5000cca25316d614           ONLINE       0     0     0
        wwn-0x5000cca2849e16ec           ONLINE       0     0     0
        wwn-0x5000cca253178e50           ONLINE       0     0     0
        wwn-0x5000cca2531bc948           ONLINE       0     0     0
        wwn-0x5000cca2531d7500           ONLINE       0     0     0
      raidz3-2                           ONLINE       0     0     0
        wwn-0x5000cca2530a4188           ONLINE       0     0     0
        wwn-0x5000cca2530a461c           ONLINE       0     0     0
        wwn-0x5000cca2530a4868           ONLINE       0     0     0
        wwn-0x5000cca2530a4918           ONLINE       0     0     0
        wwn-0x5000cca2530a4bc4           ONLINE       0     0     0
        wwn-0x5000cca2530a58a4           ONLINE       0     0     0
        dm-uuid-mpath-35000cca2530a63f4  ONLINE       0     0     0
        wwn-0x5000cca2530a6adc           ONLINE       0     0     0
        wwn-0x5000cca2530a6f3c           ONLINE       0     0     0
        wwn-0x5000cca2530a7108           ONLINE       0     0     0
        wwn-0x5000cca2530a7130           ONLINE       0     0     0
        wwn-0x5000cca2530a7288           ONLINE       0     0     0
        wwn-0x5000cca2530a7408           ONLINE       0     0     0
        wwn-0x5000cca2530a7428           ONLINE       0     0     0
        wwn-0x5000cca2530a7494           ONLINE       0     0     0
      raidz3-3                           ONLINE       0     0     0
        wwn-0x5000cca253032d58           ONLINE       0     0     0
        scsi-35000cca253075af0           ONLINE       0     0     0
        wwn-0x5000cca253084ee4           ONLINE       0     0     0
        wwn-0x5000cca2aa29bc90           ONLINE       0     0     0
        wwn-0x5000cca25308a028           ONLINE       0     0     0
        wwn-0x5000cca253090cbc           ONLINE       0     0     0
        wwn-0x5000cca253091b9c           ONLINE       0     0     0
        wwn-0x5000cca2530925bc           ONLINE       0     0     0
        wwn-0x5000cca253093758           ONLINE       0     0     0
        wwn-0x5000cca25309e6d8           ONLINE       0     0     0
        wwn-0x5000cca2530a0dd0           ONLINE       0     0     0
        wwn-0x5000cca2530a0f64           ONLINE       0     0     0
        wwn-0x5000cca2530a1140           ONLINE       0     0     0
        wwn-0x5000cca2530a21c4           ONLINE       0     0     0
        wwn-0x5000cca2530a23b4           ONLINE       0     0     0
    special 
      mirror-4                           ONLINE       0     0     0
        zfs-0ef5386f750f66d9             ONLINE       0     0     0
        zfs-a2c8272a11a49ac3             ONLINE       0     0     0
      mirror-6                           ONLINE       0     0     0
        zfs-ef648f8d22311c93             ONLINE       0     0     0
        zfs-9f637799f030adc6             ONLINE       0     0     0
    logs    
      system-slog                        ONLINE       0     0     0
    cache
      nvme4n1                            ONLINE       0     0     0
      nvme5n1                            ONLINE       0     0     0

errors: 1 data errors, use '-v' for a list

And a subsequent zpool status -v also indicated an error, but did not show the file (nor a hex address pointer that I have sometimes seen),

[root@zfs4 ~]# zpool status -v
  pool: home3
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 4 days 11:37:41 with 0 errors on Fri Nov  8 18:56:33 2024
config:

    NAME                                 STATE     READ WRITE CKSUM
    home3                                ONLINE       0     0     0
      raidz3-0                           ONLINE       0     0     0
        wwn-0x5000cca2531dd934           ONLINE       0     0     0
        wwn-0x5000cca2531e2a94           ONLINE       0     0     0
        wwn-0x5000cca29187c34c           ONLINE       0     0     0
        wwn-0x5000cca2b8482d4c           ONLINE       0     0     0
        wwn-0x5000cca26fcec890           ONLINE       0     0     0
        wwn-0x5000cca2531e5f6c           ONLINE       0     0     0
        wwn-0x5000cca27003bb00           ONLINE       0     0     0
        wwn-0x5000cca2531e84ec           ONLINE       0     0     0
        wwn-0x5000cca2531e868c           ONLINE       0     0     0
        wwn-0x5000cca2531e87d8           ONLINE       0     0     0
        wwn-0x5000cca2531e9750           ONLINE       0     0     0
        wwn-0x5000cca2b0d97f08           ONLINE       0     0     0
        wwn-0x5000cca2531e9f48           ONLINE       0     0     0
        wwn-0x5000cca2531eb96c           ONLINE       0     0     0
        wwn-0x5000cca2531ec858           ONLINE       0     0     0
      raidz3-1                           ONLINE       0     0     0
        wwn-0x5000cca2530aa110           ONLINE       0     0     0
        wwn-0x5000cca2530aa424           ONLINE       0     0     0
        wwn-0x5000cca2530aacb4           ONLINE       0     0     0
        wwn-0x5000cca2530e297c           ONLINE       0     0     0
        wwn-0x5000cca2530f661c           ONLINE       0     0     0
        wwn-0x5000cca291abaed0           ONLINE       0     0     0
        wwn-0x5000cca253123c08           ONLINE       0     0     0
        wwn-0x5000cca253158878           ONLINE       0     0     0
        wwn-0x5000cca253168af4           ONLINE       0     0     0
        wwn-0x5000cca2b0044388           ONLINE       0     0     0
        wwn-0x5000cca25316d614           ONLINE       0     0     0
        wwn-0x5000cca2849e16ec           ONLINE       0     0     0
        wwn-0x5000cca253178e50           ONLINE       0     0     0
        wwn-0x5000cca2531bc948           ONLINE       0     0     0
        wwn-0x5000cca2531d7500           ONLINE       0     0     0
      raidz3-2                           ONLINE       0     0     0
        wwn-0x5000cca2530a4188           ONLINE       0     0     0
        wwn-0x5000cca2530a461c           ONLINE       0     0     0
        wwn-0x5000cca2530a4868           ONLINE       0     0     0
        wwn-0x5000cca2530a4918           ONLINE       0     0     0
        wwn-0x5000cca2530a4bc4           ONLINE       0     0     0
        wwn-0x5000cca2530a58a4           ONLINE       0     0     0
        dm-uuid-mpath-35000cca2530a63f4  ONLINE       0     0     0
        wwn-0x5000cca2530a6adc           ONLINE       0     0     0
        wwn-0x5000cca2530a6f3c           ONLINE       0     0     0
        wwn-0x5000cca2530a7108           ONLINE       0     0     0
        wwn-0x5000cca2530a7130           ONLINE       0     0     0
        wwn-0x5000cca2530a7288           ONLINE       0     0     0
        wwn-0x5000cca2530a7408           ONLINE       0     0     0
        wwn-0x5000cca2530a7428           ONLINE       0     0     0
        wwn-0x5000cca2530a7494           ONLINE       0     0     0
      raidz3-3                           ONLINE       0     0     0
        wwn-0x5000cca253032d58           ONLINE       0     0     0
        scsi-35000cca253075af0           ONLINE       0     0     0
        wwn-0x5000cca253084ee4           ONLINE       0     0     0
        wwn-0x5000cca2aa29bc90           ONLINE       0     0     0
        wwn-0x5000cca25308a028           ONLINE       0     0     0
        wwn-0x5000cca253090cbc           ONLINE       0     0     0
        wwn-0x5000cca253091b9c           ONLINE       0     0     0
        wwn-0x5000cca2530925bc           ONLINE       0     0     0
        wwn-0x5000cca253093758           ONLINE       0     0     0
        wwn-0x5000cca25309e6d8           ONLINE       0     0     0
        wwn-0x5000cca2530a0dd0           ONLINE       0     0     0
        wwn-0x5000cca2530a0f64           ONLINE       0     0     0
        wwn-0x5000cca2530a1140           ONLINE       0     0     0
        wwn-0x5000cca2530a21c4           ONLINE       0     0     0
        wwn-0x5000cca2530a23b4           ONLINE       0     0     0
    special 
      mirror-4                           ONLINE       0     0     0
        zfs-0ef5386f750f66d9             ONLINE       0     0     0
        zfs-a2c8272a11a49ac3             ONLINE       0     0     0
      mirror-6                           ONLINE       0     0     0
        zfs-ef648f8d22311c93             ONLINE       0     0     0
        zfs-9f637799f030adc6             ONLINE       0     0     0
    logs    
      system-slog                        ONLINE       0     0     0
    cache
      nvme4n1                            ONLINE       0     0     0
      nvme5n1                            ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

Howwever, another zpool status indicated there are no errors (which I believe is the truth),

[root@zfs4 ~]# zpool status
  pool: home3
 state: ONLINE
  scan: scrub repaired 0B in 4 days 11:37:41 with 0 errors on Fri Nov  8 18:56:33 2024
config:

    NAME                                 STATE     READ WRITE CKSUM
    home3                                ONLINE       0     0     0
      raidz3-0                           ONLINE       0     0     0
        wwn-0x5000cca2531dd934           ONLINE       0     0     0
        wwn-0x5000cca2531e2a94           ONLINE       0     0     0
        wwn-0x5000cca29187c34c           ONLINE       0     0     0
        wwn-0x5000cca2b8482d4c           ONLINE       0     0     0
        wwn-0x5000cca26fcec890           ONLINE       0     0     0
        wwn-0x5000cca2531e5f6c           ONLINE       0     0     0
        wwn-0x5000cca27003bb00           ONLINE       0     0     0
        wwn-0x5000cca2531e84ec           ONLINE       0     0     0
        wwn-0x5000cca2531e868c           ONLINE       0     0     0
        wwn-0x5000cca2531e87d8           ONLINE       0     0     0
        wwn-0x5000cca2531e9750           ONLINE       0     0     0
        wwn-0x5000cca2b0d97f08           ONLINE       0     0     0
        wwn-0x5000cca2531e9f48           ONLINE       0     0     0
        wwn-0x5000cca2531eb96c           ONLINE       0     0     0
        wwn-0x5000cca2531ec858           ONLINE       0     0     0
      raidz3-1                           ONLINE       0     0     0
        wwn-0x5000cca2530aa110           ONLINE       0     0     0
        wwn-0x5000cca2530aa424           ONLINE       0     0     0
        wwn-0x5000cca2530aacb4           ONLINE       0     0     0
        wwn-0x5000cca2530e297c           ONLINE       0     0     0
        wwn-0x5000cca2530f661c           ONLINE       0     0     0
        wwn-0x5000cca291abaed0           ONLINE       0     0     0
        wwn-0x5000cca253123c08           ONLINE       0     0     0
        wwn-0x5000cca253158878           ONLINE       0     0     0
        wwn-0x5000cca253168af4           ONLINE       0     0     0
        wwn-0x5000cca2b0044388           ONLINE       0     0     0
        wwn-0x5000cca25316d614           ONLINE       0     0     0
        wwn-0x5000cca2849e16ec           ONLINE       0     0     0
        wwn-0x5000cca253178e50           ONLINE       0     0     0
        wwn-0x5000cca2531bc948           ONLINE       0     0     0
        wwn-0x5000cca2531d7500           ONLINE       0     0     0
      raidz3-2                           ONLINE       0     0     0
        wwn-0x5000cca2530a4188           ONLINE       0     0     0
        wwn-0x5000cca2530a461c           ONLINE       0     0     0
        wwn-0x5000cca2530a4868           ONLINE       0     0     0
        wwn-0x5000cca2530a4918           ONLINE       0     0     0
        wwn-0x5000cca2530a4bc4           ONLINE       0     0     0
        wwn-0x5000cca2530a58a4           ONLINE       0     0     0
        dm-uuid-mpath-35000cca2530a63f4  ONLINE       0     0     0
        wwn-0x5000cca2530a6adc           ONLINE       0     0     0
        wwn-0x5000cca2530a6f3c           ONLINE       0     0     0
        wwn-0x5000cca2530a7108           ONLINE       0     0     0
        wwn-0x5000cca2530a7130           ONLINE       0     0     0
        wwn-0x5000cca2530a7288           ONLINE       0     0     0
        wwn-0x5000cca2530a7408           ONLINE       0     0     0
        wwn-0x5000cca2530a7428           ONLINE       0     0     0
        wwn-0x5000cca2530a7494           ONLINE       0     0     0
      raidz3-3                           ONLINE       0     0     0
        wwn-0x5000cca253032d58           ONLINE       0     0     0
        scsi-35000cca253075af0           ONLINE       0     0     0
        wwn-0x5000cca253084ee4           ONLINE       0     0     0
        wwn-0x5000cca2aa29bc90           ONLINE       0     0     0
        wwn-0x5000cca25308a028           ONLINE       0     0     0
        wwn-0x5000cca253090cbc           ONLINE       0     0     0
        wwn-0x5000cca253091b9c           ONLINE       0     0     0
        wwn-0x5000cca2530925bc           ONLINE       0     0     0
        wwn-0x5000cca253093758           ONLINE       0     0     0
        wwn-0x5000cca25309e6d8           ONLINE       0     0     0
        wwn-0x5000cca2530a0dd0           ONLINE       0     0     0
        wwn-0x5000cca2530a0f64           ONLINE       0     0     0
        wwn-0x5000cca2530a1140           ONLINE       0     0     0
        wwn-0x5000cca2530a21c4           ONLINE       0     0     0
        wwn-0x5000cca2530a23b4           ONLINE       0     0     0
    special 
      mirror-4                           ONLINE       0     0     0
        zfs-0ef5386f750f66d9             ONLINE       0     0     0
        zfs-a2c8272a11a49ac3             ONLINE       0     0     0
      mirror-6                           ONLINE       0     0     0
        zfs-ef648f8d22311c93             ONLINE       0     0     0
        zfs-9f637799f030adc6             ONLINE       0     0     0
    logs    
      system-slog                        ONLINE       0     0     0
    cache
      nvme4n1                            ONLINE       0     0     0
      nvme5n1                            ONLINE       0     0     0

errors: No known data errors

Include any warning/errors/backtraces from the system logs

During and after the zpool scrub no low-level system/device error messages where recorded.