skunkr commented 5 years ago

Hi Everyone, I have this very big issue with all the configured drives randomly getting into errors. The environment is as follows:

6 virtual machines running on top of redhat virtualisation ( rhhci )
each vm is running Centos7.5.1804
vdo version is 6.1.1.125 on all the vms
each of the vms has 10 x enterprise grade, spinning, 12gb/s, 3.6TB drives passthrough mode from the physical hosts.
on each of the vms, the vdo drives where configured as: logical volume 35TB, index size 4GB, sync mode
underlying block device is actually a raid0 volume configured with cache and write back
if I used the vm drives without vdo, everything work fine - so not a passthrough issue
each of the vdo volumes was formated with xfs with discard, and also mounted by using systemd services as per the given example.

But at some point they are randomly dying.... Below, some example errors regarding one volume ( "vsdh" in this case ):

vdostats

Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vsda 3906249728 48084728 3858165000 1% 96% /dev/mapper/vsdb 3906249728 48178260 3858071468 1% 92% /dev/mapper/vsdc 3906249728 48165820 3858083908 1% 93% /dev/mapper/vsdd 3906249728 48316504 3857933224 1% 87% /dev/mapper/vsde 3906249728 48304600 3857945128 1% 87% /dev/mapper/vsdf 3906249728 48176624 3858073104 1% 92% /dev/mapper/vsdg 3906249728 48211872 3858037856 1% 90% /dev/mapper/vsdh 3906249728 N/A N/A N/A N/A /dev/mapper/vsdi 3906249728 48192572 3858057156 1% 91% /dev/mapper/vsdj 3906249728 48133656 3858116072 1% 94%

cat /var/log/messages | grep vdo

Jul 25 00:37:22 hdp-datanode-5 kernel: kvdo8:logQ0: Expected page 11058242 but got page 11058240 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 00:37:23 hdp-datanode-5 kernel: uds: kvdo7:dedupeQ: read index page map, last update 0 Jul 25 00:37:23 hdp-datanode-5 kernel: uds: kvdo7:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 25 00:37:24 hdp-datanode-5 kernel: uds: kvdo8:dedupeQ: read index page map, last update 0 Jul 25 00:37:24 hdp-datanode-5 kernel: uds: kvdo8:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 25 00:37:25 hdp-datanode-5 kernel: uds: kvdo9:dedupeQ: read index page map, last update 0 Jul 25 00:37:25 hdp-datanode-5 kernel: uds: kvdo9:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 25 06:19:34 hdp-datanode-5 kernel: kvdo4:logQ0: Expected page 11058137 but got page 11058019 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 06:22:36 hdp-datanode-5 kernel: kvdo8:logQ0: Expected page 11058933 but got page 11058982 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 06:22:36 hdp-datanode-5 kernel: kvdo8:logQ0: Expected page 11058940 but got page 11058934 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 06:22:36 hdp-datanode-5 kernel: kvdo8:logQ0: Completing read VIO for LBN 2415919097 with error after getMappedBlock: kvdo: Corrupt or incorrect page (2057) Jul 25 06:22:36 hdp-datanode-5 kernel: kvdo8:cpuQ0: Jul 25 06:22:36 hdp-datanode-5 kernel: kvdo8:journalQ: Jul 25 06:22:36 hdp-datanode-5 kernel: : kvdo: Corrupt or incorrect page (2057) Jul 25 06:22:36 hdp-datanode-5 kernel: mapToSystemError: mapping internal status code 2057 (kvdo: VDO_BAD_PAGE: kvdo: Corrupt or incorrect page) to EIO Jul 25 06:23:10 hdp-datanode-5 kernel: kvdo2:logQ0: Expected page 11058124 but got page 11058120 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 06:24:41 hdp-datanode-5 kernel: kvdo7:logQ0: Expected page 11058095 but got page 11058090 instead: kvdo: Corrupt or incorrect page (2057) Jul 25 06:25:19 hdp-datanode-5 kernel: kvdo0:logQ0: Expected page 11058216 but got page 11058208 instead: kvdo: Corrupt or incorrect page (2057)

vdo status -n vsdh

VDO status: Date: '2019-07-25 07:23:29+00:00' Node: hdp-datanode-5.mptjo.internal Kernel module: Loaded: true Name: kvdo Version information: kvdo version: 6.1.1.125 Configuration: File: /etc/vdoconf.yml Last modified: '2019-07-24 17:53:44' VDOs: vsdh: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: enabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 75161927680 vdo /dev/sdh read-only - online online 12001210 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 4 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 35T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314e1c024c6fa21cdf899cc VDO statistics: /dev/mapper/vsdh: 1K-blocks: 3906249728 1K-blocks available: N/A 1K-blocks used: N/A 512 byte emulation: false KVDO module bios used: 372860 KVDO module bytes used: 51372628896 KVDO module peak bio count: 373148 KVDO module peak bytes used: 51372817808 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 0 bios acknowledged partial write: 0 bios acknowledged read: 780 bios acknowledged write: 1016 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 780 bios in write: 1016 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 278 bios journal discard: 0 bios journal flush: 278 bios journal fua: 278 bios journal read: 0 bios journal write: 278 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 293 bios meta completed write: 403 bios meta discard: 0 bios meta flush: 282 bios meta fua: 280 bios meta read: 293 bios meta write: 403 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 219 bios out completed write: 1012 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 219 bios out write: 1012 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 21 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 21 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 9 block map discard required: 0 block map failed pages: 0 block map failed reads: 1 block map failed writes: 0 block map fetch required: 21 block map flush count: 0 block map found in cache: 3468 block map free pages: 32748 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 21 block map pages saved: 0 block map read count: 1778 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 1711 block size: 4096 completed recovery count: 0 compressed blocks written: 55 compressed fragments in packer: 30 compressed fragments written: 662 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 30 current dedupe queries: 0 data blocks used: N/A dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 37 entries indexed: 1209 flush out: 0 instance: 8 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 278 journal blocks started: 278 journal blocks writing: 0 journal blocks written: 278 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 3422 journal entries started: 3422 journal entries writing: 0 journal entries written: 3422 logical blocks: 9395240960 logical blocks used: 522409 maximum VDO IO requests in progress: 512 maximum dedupe queries: 160 no space error count: 0 operating mode: read-only overhead blocks used: 12000831 physical blocks: 976562432 posts found: 37 posts not found: 463 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 4 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: N/A slab count: 1841 slab journal blocked count: 0 slab journal blocks written: 2 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 2 slabs opened: 1 slabs reopened: 1 updates found: 150 updates not found: 8 used percent: N/A version: 28 write amplification ratio: 1.39 write policy: sync

Could you please help me out sorting this issue ? It already became a very big blocker in moving forward with other deployments.. Thank you very much ! Leo

drckeefe commented 5 years ago

Leo,

Looking at the output of VDO stats these fields are interesting

operating mode: read-only - Of course this means the vdo volume is in read-only mode at the moment A forced rebuild will be needed to get out of this mode.
block map failed reads: 1 - I wonder if this is a read error from media or metadata was corrupted
KVDO module peak bytes used: 51372817808 - 50GB seems high for this volume - was there a larger block map configured for this volume prior?

Can you tell me the make and model of the drives in RAID0 and drives being used as cache and writeback cache? I've seen errors like this when there is a read to an empty block and instead of the drive returning zeros, it returns data from memory.

skunkr commented 5 years ago

First of all, thank you very much for helping me with this situation, it is a really good feeling knowing that there are chances to fix these issues... Some infos regarding he environment:

HW and Setup Infos

Server: ThinkSystem SR650 Raid controller: ThinkSystem RAID 930-16i 4GB Flash PCIe 12Gb Adapter Underlying drives are a mix of:

Seagate - ST4000NM0025 - 4TB, 7200RPM, 12GB/s, 128Mb cache, 512 bytes per sector
Toshiba - MG04SCA40EN - same hw specs as the seagate ( I could not associate the vdo fails with a particular hdd vendor so far - it seems to be all randomly )
The 512 bytes makes me wonder if this would be a part of the problem since vdo native is 4096, an maybe the 512 emulation should be configured.
Each server has 10 drives installed to be assigned to vms
Each server is a hypevisor node running RedHat Virtualisation ( RHHCI )
At the hypervisor level, the drives are configured as passthrough mode and given to the virtual machine.
At the moment there are six hypervisors, so 6 vms ( each vm having its own 10 drives attached ) but if I focus on one of them and have it stable, then i assume I can do the same setup for all of them.

Each of the raid0 arrays are configured as per below:

Read Policy: Read Ahead Access Policy: Read Write Default Write Policy: Always Write Back Drive Cache: Enabled I/O Policy: Cached Background Initialization: Enabled

Before creating the vdos by using cockpit UI, I have wiped each disk with "gdisk" ( maybe I should have proceed in a different way ? )
All the vdos have been formated like: mkfs.xfs -K /dev/mapper/vsda
All the vdo's are mounted as per redhat documentation, by using systemd services with "discard" option

Operations

First setup was done with the disks attached as jbod, but I have encontered some performance issues, and thought that enabling raid mode with cache will speed up the read/writes a bit - which actually happend, but didn't solved the vdo volumes stability issues...
Yesterday I have disabled compression on all of them, and also chnaged the logical size to be 3:1 ( 10TB each volume ). After this, the number of failed vdo's reduced, but still we have failed vdo's, as per the following scenario:
After performing normal writes to the mounpoint or just a system reboot, some vdo's enter read-only mode a. some of them can be fixed by stopping and starting back with --forceRebuild option b. some of the rebuilded ones can be mounted, and some cannot be mounted due to xfs filesystem errors\
I cannot see reported hardware disk fails or errors at the server level

Below, you can find the actual confs and status and I will keep the environment as it is, for easier troubleshooting. Please let me know if any other logs are helpfull in troubleshooting this situation. Again, thank you very much for helping me in sorting this out.

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 3.7T 0 disk └─vsda 253:0 0 10T 0 vdo /hadoop01/data sdb 8:16 0 3.7T 0 disk └─vsdb 253:2 0 10T 0 vdo
sdc 8:32 0 3.7T 0 disk └─vsdc 253:1 0 10T 0 vdo /hadoop03/data sdd 8:48 0 3.7T 0 disk └─vsdd 253:4 0 10T 0 vdo /hadoop04/data sde 8:64 0 3.7T 0 disk └─vsde 253:3 0 10T 0 vdo
sdf 8:80 0 3.7T 0 disk └─vsdf 253:6 0 10T 0 vdo /hadoop06/data sdg 8:96 0 3.7T 0 disk └─vsdg 253:5 0 10T 0 vdo /hadoop07/data sdh 8:112 0 3.7T 0 disk └─vsdh 253:8 0 10T 0 vdo /hadoop08/data sdi 8:128 0 3.7T 0 disk └─vsdi 253:7 0 10T 0 vdo
sdj 8:144 0 3.7T 0 disk └─vsdj 253:9 0 10T 0 vdo /hadoop10/data sr0 11:0 1 1024M 0 rom
vda 252:0 0 70G 0 disk └─vda1 252:1 0 70G 0 part /

vdostats

Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vsda 3906249728 26208420 3880041308 0% 69% /dev/mapper/vsdb 3906249728 25152932 3881096796 0% 93% /dev/mapper/vsdc 3906249728 25173216 3881076512 0% 93% /dev/mapper/vsdd 3906249728 25135564 3881114164 0% 94% /dev/mapper/vsde 3906249728 25255028 3880994700 0% 90% /dev/mapper/vsdf 3906249728 25182096 3881067632 0% 92% /dev/mapper/vsdg 3906249728 25273024 3880976704 0% 89% /dev/mapper/vsdh 3906249728 25202912 3881046816 0% 92% /dev/mapper/vsdi 3906249728 N/A N/A N/A N/A /dev/mapper/vsdj 3906249728 25339252 3880910476 0% 88%

vdo status

VDO status: Date: '2019-07-26 07:10:19+00:00' Node: hdp-datanode-1.internal Kernel module: Loaded: true Name: kvdo Version information: kvdo version: 6.1.1.125 Configuration: File: /etc/vdoconf.yml Last modified: '2019-07-25 10:51:41' VDOs: vsda: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sda normal - online offline 6552105 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0445be7ab1a VDO statistics: /dev/mapper/vsda: 1K-blocks: 3906249728 1K-blocks available: 3880041308 1K-blocks used: 26208420 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 1179 bios acknowledged partial write: 0 bios acknowledged read: 1323 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 1323 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 30 bios journal discard: 0 bios journal flush: 30 bios journal fua: 30 bios journal read: 0 bios journal write: 30 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 396 bios meta completed write: 95 bios meta discard: 0 bios meta flush: 31 bios meta fua: 31 bios meta read: 396 bios meta write: 95 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 1179 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 1179 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 14 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 14 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 12 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 14 block map flush count: 0 block map found in cache: 2332 block map free pages: 32754 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 14 block map pages saved: 0 block map read count: 1834 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 313631 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 318729 flush out: 0 instance: 0 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 30 journal blocks started: 30 journal blocks writing: 0 journal blocks written: 30 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 1027410 maximum VDO IO requests in progress: 444 maximum dedupe queries: 277 no space error count: 0 operating mode: normal overhead blocks used: 6238474 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 69 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 2 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.19 write policy: sync vsdb: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdb normal - online offline 6288233 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0b504ad5ed6 VDO statistics: /dev/mapper/vsdb: 1K-blocks: 3906249728 1K-blocks available: 3881096796 1K-blocks used: 25152932 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 112 bios acknowledged partial write: 0 bios acknowledged read: 365 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 365 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 24 bios journal discard: 0 bios journal flush: 24 bios journal fua: 24 bios journal read: 0 bios journal write: 24 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 91 bios meta discard: 0 bios meta flush: 26 bios meta fua: 25 bios meta read: 260 bios meta write: 91 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 624 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 624 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1884 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 876 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 1021 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 50190 dedupe advice stale: 3 dedupe advice timeouts: 0 dedupe advice valid: 509 entries indexed: 53144 flush out: 0 instance: 2 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 24 journal blocks started: 24 journal blocks writing: 0 journal blocks written: 24 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 2042 journal entries started: 2042 journal entries writing: 0 journal entries written: 2042 logical blocks: 2684354560 logical blocks used: 768877 maximum VDO IO requests in progress: 512 maximum dedupe queries: 207 no space error count: 0 operating mode: normal overhead blocks used: 6238043 physical blocks: 976562432 posts found: 512 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 93 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 1 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 1 slabs opened: 1 slabs reopened: 1 updates found: 3 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.18 write policy: sync vsdc: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdc normal - online offline 6293304 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0e54d1017cf VDO statistics: /dev/mapper/vsdc: 1K-blocks: 3906249728 1K-blocks available: 3881076512 1K-blocks used: 25173216 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 390 bios acknowledged partial write: 0 bios acknowledged read: 643 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 643 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 10 bios journal discard: 0 bios journal flush: 10 bios journal fua: 10 bios journal read: 0 bios journal write: 10 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 75 bios meta discard: 0 bios meta flush: 11 bios meta fua: 11 bios meta read: 260 bios meta write: 75 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 390 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 390 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1653 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1154 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 55148 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 59201 flush out: 0 instance: 1 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 10 journal blocks started: 10 journal blocks writing: 0 journal blocks written: 10 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 793191 maximum VDO IO requests in progress: 516 maximum dedupe queries: 512 no space error count: 0 operating mode: normal overhead blocks used: 6238156 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 93 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdd: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdd normal - online offline 6283891 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f05e840f422d VDO statistics: /dev/mapper/vsdd: 1K-blocks: 3906249728 1K-blocks available: 3881114164 1K-blocks used: 25135564 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 356 bios acknowledged partial write: 0 bios acknowledged read: 609 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 609 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 9 bios journal discard: 0 bios journal flush: 9 bios journal fua: 9 bios journal read: 0 bios journal write: 9 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 74 bios meta discard: 0 bios meta flush: 10 bios meta fua: 10 bios meta read: 260 bios meta write: 74 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 356 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 356 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1619 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1120 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 45744 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 49805 flush out: 0 instance: 4 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 9 journal blocks started: 9 journal blocks writing: 0 journal blocks written: 9 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 765861 maximum VDO IO requests in progress: 516 maximum dedupe queries: 356 no space error count: 0 operating mode: normal overhead blocks used: 6238147 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 94 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.14 write policy: sync vsde: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sde normal - online offline 6313757 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0191bd587b0 VDO statistics: /dev/mapper/vsde: 1K-blocks: 3906249728 1K-blocks available: 3880994700 1K-blocks used: 25255028 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 618 bios acknowledged partial write: 0 bios acknowledged read: 871 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 871 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 48 bios journal discard: 0 bios journal flush: 48 bios journal fua: 48 bios journal read: 0 bios journal write: 48 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 115 bios meta discard: 0 bios meta flush: 50 bios meta fua: 49 bios meta read: 260 bios meta write: 115 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 1130 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 1130 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 2391 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1382 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 1022 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 75649 dedupe advice stale: 2 dedupe advice timeouts: 0 dedupe advice valid: 510 entries indexed: 78489 flush out: 0 instance: 3 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 48 journal blocks started: 48 journal blocks writing: 0 journal blocks written: 48 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 2044 journal entries started: 2044 journal entries writing: 0 journal entries written: 2044 logical blocks: 2684354560 logical blocks used: 790218 maximum VDO IO requests in progress: 512 maximum dedupe queries: 210 no space error count: 0 operating mode: normal overhead blocks used: 6238108 physical blocks: 976562432 posts found: 512 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 90 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 1 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 1 slabs opened: 1 slabs reopened: 1 updates found: 2 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.22 write policy: sync vsdf: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdf normal - online offline 6295524 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f07aacef5dd4 VDO statistics: /dev/mapper/vsdf: 1K-blocks: 3906249728 1K-blocks available: 3881067632 1K-blocks used: 25182096 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 492 bios acknowledged partial write: 0 bios acknowledged read: 745 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 745 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 11 bios journal discard: 0 bios journal flush: 11 bios journal fua: 11 bios journal read: 0 bios journal write: 11 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 76 bios meta discard: 0 bios meta flush: 12 bios meta fua: 12 bios meta read: 260 bios meta write: 76 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 492 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 492 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1755 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1256 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 57396 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 61820 flush out: 0 instance: 6 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 11 journal blocks started: 11 journal blocks writing: 0 journal blocks written: 11 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 779861 maximum VDO IO requests in progress: 512 maximum dedupe queries: 175 no space error count: 0 operating mode: normal overhead blocks used: 6238128 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 92 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdg: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdg normal - online offline 6318256 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6eeacfbf3d2b0 VDO statistics: /dev/mapper/vsdg: 1K-blocks: 3906249728 1K-blocks available: 3880976704 1K-blocks used: 25273024 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 398 bios acknowledged partial write: 0 bios acknowledged read: 651 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 651 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 27 bios journal discard: 0 bios journal flush: 27 bios journal fua: 27 bios journal read: 0 bios journal write: 27 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 92 bios meta discard: 0 bios meta flush: 28 bios meta fua: 28 bios meta read: 260 bios meta write: 92 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 398 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 398 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1661 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1162 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 80074 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 84167 flush out: 0 instance: 5 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 27 journal blocks started: 27 journal blocks writing: 0 journal blocks written: 27 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 796002 maximum VDO IO requests in progress: 446 maximum dedupe queries: 297 no space error count: 0 operating mode: normal overhead blocks used: 6238182 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 89 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.18 write policy: sync vsdh: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdh normal - online offline 6300728 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f09cdf19554b VDO statistics: /dev/mapper/vsdh: 1K-blocks: 3906249728 1K-blocks available: 3881046816 1K-blocks used: 25202912 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 350 bios acknowledged partial write: 0 bios acknowledged read: 603 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 603 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 14 bios journal discard: 0 bios journal flush: 14 bios journal fua: 14 bios journal read: 0 bios journal write: 14 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 258 bios meta completed write: 79 bios meta discard: 0 bios meta flush: 15 bios meta fua: 15 bios meta read: 258 bios meta write: 79 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 350 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 350 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 11 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 11 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 9 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 11 block map flush count: 0 block map found in cache: 1613 block map free pages: 32757 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 11 block map pages saved: 0 block map read count: 1112 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 62608 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 63012 flush out: 0 instance: 8 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 14 journal blocks started: 14 journal blocks writing: 0 journal blocks written: 14 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 815932 maximum VDO IO requests in progress: 468 maximum dedupe queries: 0 no space error count: 0 operating mode: normal overhead blocks used: 6238120 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 92 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdi: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdi read-only - online offline 6236664 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f126addf5b03 VDO statistics: /dev/mapper/vsdi: 1K-blocks: 3906249728 1K-blocks available: N/A 1K-blocks used: N/A 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 19 bios acknowledged partial write: 0 bios acknowledged read: 263 bios acknowledged write: 0 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 263 bios in write: 0 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 0 bios journal discard: 0 bios journal flush: 0 bios journal fua: 0 bios journal read: 0 bios journal write: 0 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 19 bios meta completed write: 0 bios meta discard: 0 bios meta flush: 0 bios meta fua: 0 bios meta read: 19 bios meta write: 0 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 19 bios out completed write: 0 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 19 bios out write: 0 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 3 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 3 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 2 block map dirty pages: 0 block map discard required: 0 block map failed pages: 0 block map failed reads: 1 block map failed writes: 0 block map fetch required: 3 block map flush count: 0 block map found in cache: 259 block map free pages: 32766 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 3 block map pages saved: 0 block map read count: 262 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 0 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: N/A dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 72140 flush out: 0 instance: 7 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 0 journal blocks started: 0 journal blocks writing: 0 journal blocks written: 0 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 0 journal entries started: 0 journal entries writing: 0 journal entries written: 0 logical blocks: 2684354560 logical blocks used: 852316 maximum VDO IO requests in progress: 30 maximum dedupe queries: 0 no space error count: 0 operating mode: read-only overhead blocks used: 6238222 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: N/A slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 0 slabs reopened: 0 updates found: 0 updates not found: 0 used percent: N/A version: 28 write amplification ratio: 0.0 write policy: sync vsdj: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdj normal - online offline 6334813 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f02e3c597e3b VDO statistics: /dev/mapper/vsdj: 1K-blocks: 3906249728 1K-blocks available: 3880910476 1K-blocks used: 25339252 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 354 bios acknowledged partial write: 0 bios acknowledged read: 607 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 607 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 11 bios journal discard: 0 bios journal flush: 11 bios journal fua: 11 bios journal read: 0 bios journal write: 11 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 76 bios meta discard: 0 bios meta flush: 12 bios meta fua: 12 bios meta read: 260 bios meta write: 76 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 354 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 354 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1617 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1118 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 96576 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 96941 flush out: 0 instance: 9 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 11 journal blocks started: 11 journal blocks writing: 0 journal blocks written: 11 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 846871 maximum VDO IO requests in progress: 512 maximum dedupe queries: 0 no space error count: 0 operating mode: normal overhead blocks used: 6238237 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 88 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync

Some of the latest logs

cat /var/log/messages | grep "kernel: XFS"

Jul 26 12:25:28 hdp-datanode-1 kernel: XFS (vda1): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-1): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-0): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-6): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-8): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-9): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-5): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): metadata I/O error: block 0x280000010 ("xlog_bread_noalign") error 5 numblks 8 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): empty log check failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): log mount/recovery failed: error -5 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): log mount failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-4): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-5): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): Corruption warning: Metadata has LSN (1:2736) ahead of current LSN (1:120). Please unmount and run xfs_repair (>= v4.3) to resolve. Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): log mount/recovery failed: error -22 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): log mount failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-1): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-4): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-6): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-8): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-0): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): Corruption warning: Metadata has LSN (1:4824) ahead of current LSN (1:1984). Please unmount and run xfs_repair (>= v4.3) to resolve. Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-9): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): log mount/recovery failed: error -22 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): log mount failed

cat /var/log/messages | grep vdo

Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:journalQ: VDO commencing normal operation Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: Jul 26 05:55:38 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: Jul 26 05:55:38 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: device 'vsdb' started Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: resuming device 'vsdb' Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: device 'vsdb' resumed Jul 26 05:55:39 hdp-datanode-1 UDS/vdodmeventd[3694]: INFO (vdodmeventd/3694) VDO device vsdb is now registered with dmeventd for monitoring Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: starting device 'vsde' Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: Using write policy sync automatically. Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:39 hdp-datanode-1 kernel: uds: kvdo0:dedupeQ: read index page map, last update 0 Jul 26 05:55:39 hdp-datanode-1 kernel: uds: kvdo0:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:journalQ: VDO commencing normal operation Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: device 'vsde' started Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: resuming device 'vsde' Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: device 'vsde' resumed Jul 26 05:55:41 hdp-datanode-1 UDS/vdodmeventd[3790]: INFO (vdodmeventd/3790) VDO device vsde is now registered with dmeventd for monitoring Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: starting device 'vsdd' Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: Using write policy sync automatically. Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo1:dedupeQ: read index page map, last update 0 Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo1:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:42 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: read index page map, last update 0 Jul 26 05:55:42 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:journalQ: VDO commencing normal operation Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: Jul 26 05:55:43 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: Jul 26 05:55:43 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: device 'vsdd' started Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: resuming device 'vsdd' Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: device 'vsdd' resumed Jul 26 05:55:43 hdp-datanode-1 UDS/vdodmeventd[3926]: INFO (vdodmeventd/3926) VDO device vsdd is now registered with dmeventd for monitoring Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: starting device 'vsdg' Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: Using write policy sync automatically. Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: read index page map, last update 0 Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo5:journalQ: VDO commencing normal operation Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo5:dmsetup: Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo5:dedupeQ: Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo5:dedupeQ: Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo5:dmsetup: device 'vsdg' started Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo5:dmsetup: resuming device 'vsdg' Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo5:dmsetup: device 'vsdg' resumed Jul 26 05:55:45 hdp-datanode-1 UDS/vdodmeventd[4056]: INFO (vdodmeventd/4056) VDO device vsdg is now registered with dmeventd for monitoring Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo6:dmsetup: starting device 'vsdf' Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo6:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo6:dmsetup: Using write policy sync automatically. Jul 26 05:55:45 hdp-datanode-1 kernel: kvdo6:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:47 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: read index page map, last update 0 Jul 26 05:55:47 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo6:journalQ: VDO commencing normal operation Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo6:dmsetup: Jul 26 05:55:48 hdp-datanode-1 kernel: uds: kvdo6:dedupeQ: Jul 26 05:55:48 hdp-datanode-1 kernel: uds: kvdo6:dedupeQ: Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo6:dmsetup: device 'vsdf' started Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo6:dmsetup: resuming device 'vsdf' Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo6:dmsetup: device 'vsdf' resumed Jul 26 05:55:48 hdp-datanode-1 UDS/vdodmeventd[4203]: INFO (vdodmeventd/4203) VDO device vsdf is now registered with dmeventd for monitoring Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo7:dmsetup: starting device 'vsdi' Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo7:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo7:dmsetup: Using write policy sync automatically. Jul 26 05:55:48 hdp-datanode-1 kernel: kvdo7:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo7:dmsetup: Jul 26 05:55:49 hdp-datanode-1 kernel: uds: kvdo7:dedupeQ: Jul 26 05:55:49 hdp-datanode-1 kernel: uds: kvdo7:dedupeQ: Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo7:dmsetup: device 'vsdi' started Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo7:dmsetup: resuming device 'vsdi' Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo7:dmsetup: device 'vsdi' resumed Jul 26 05:55:49 hdp-datanode-1 UDS/vdodmeventd[4259]: INFO (vdodmeventd/4259) VDO device vsdi is now registered with dmeventd for monitoring Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo8:dmsetup: starting device 'vsdh' Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo8:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo8:dmsetup: Using write policy sync automatically. Jul 26 05:55:49 hdp-datanode-1 kernel: kvdo8:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:50 hdp-datanode-1 kernel: uds: kvdo5:dedupeQ: read index page map, last update 0 Jul 26 05:55:50 hdp-datanode-1 kernel: uds: kvdo5:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo8:journalQ: VDO commencing normal operation Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo8:dmsetup: Jul 26 05:55:51 hdp-datanode-1 kernel: uds: kvdo8:dedupeQ: Jul 26 05:55:51 hdp-datanode-1 kernel: uds: kvdo8:dedupeQ: Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo8:dmsetup: device 'vsdh' started Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo8:dmsetup: resuming device 'vsdh' Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo8:dmsetup: device 'vsdh' resumed Jul 26 05:55:51 hdp-datanode-1 UDS/vdodmeventd[4339]: INFO (vdodmeventd/4339) VDO device vsdh is now registered with dmeventd for monitoring Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo9:dmsetup: starting device 'vsdj' Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo9:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo9:dmsetup: Using write policy sync automatically. Jul 26 05:55:51 hdp-datanode-1 kernel: kvdo9:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:52 hdp-datanode-1 kernel: uds: kvdo6:dedupeQ: read index page map, last update 0 Jul 26 05:55:52 hdp-datanode-1 kernel: uds: kvdo6:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:53 hdp-datanode-1 kernel: uds: kvdo7:dedupeQ: read index page map, last update 0 Jul 26 05:55:53 hdp-datanode-1 kernel: uds: kvdo7:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo9:journalQ: VDO commencing normal operation Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo9:dmsetup: Jul 26 05:55:54 hdp-datanode-1 kernel: uds: kvdo9:dedupeQ: Jul 26 05:55:54 hdp-datanode-1 kernel: uds: kvdo9:dedupeQ: Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo9:dmsetup: device 'vsdj' started Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo9:dmsetup: resuming device 'vsdj' Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo9:dmsetup: device 'vsdj' resumed Jul 26 05:55:54 hdp-datanode-1 UDS/vdodmeventd[4421]: INFO (vdodmeventd/4421) VDO device vsdj is now registered with dmeventd for monitoring Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsda Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 0 volume is ready at /dev/mapper/vsda Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdc Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 1 volume is ready at /dev/mapper/vsdc Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdb Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 2 volume is ready at /dev/mapper/vsdb Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsde Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 3 volume is ready at /dev/mapper/vsde Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdd Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 4 volume is ready at /dev/mapper/vsdd Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdg Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 5 volume is ready at /dev/mapper/vsdg Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdf Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 6 volume is ready at /dev/mapper/vsdf Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdi Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 7 volume is ready at /dev/mapper/vsdi Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdh Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 8 volume is ready at /dev/mapper/vsdh Jul 26 05:55:54 hdp-datanode-1 vdo: Starting VDO vsdj Jul 26 05:55:54 hdp-datanode-1 vdo: VDO instance 9 volume is ready at /dev/mapper/vsdj Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo7:logQ0: Expected page 5496303 but got page 5496294 instead: kvdo: Corrupt or incorrect page (2057) Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo7:logQ0: Completing read VIO for LBN 1342177282 with error after getMappedBlock: kvdo: Corrupt or incorrect page (2057) Jul 26 05:55:54 hdp-datanode-1 kernel: kvdo7:cpuQ1: mapToSystemError: mapping internal status code 2057 (kvdo: VDO_BAD_PAGE: kvdo: Corrupt or incorrect page) to EIO Jul 26 05:55:56 hdp-datanode-1 kernel: uds: kvdo8:dedupeQ: read index page map, last update 0 Jul 26 05:55:56 hdp-datanode-1 kernel: uds: kvdo8:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:59 hdp-datanode-1 kernel: uds: kvdo9:dedupeQ: read index page map, last update 0 Jul 26 05:55:59 hdp-datanode-1 kernel: uds: kvdo9:dedupeQ: index_0: loaded index from chapter 0 through chapter 0

drckeefe commented 5 years ago

Thank you for sending this information. I have some questions inline below.

On Fri, Jul 26, 2019 at 3:44 AM skunkr notifications@github.com wrote:

First of all, thank you very much for helping me with this situation, it is a really good feeling knowing that there are chances to fix these issues... Some infos regarding he environment: HW and Setup Infos

Server: ThinkSystem SR650 Raid controller: ThinkSystem RAID 930-16i 4GB Flash PCIe 12Gb Adapter Underlying drives are a mix of:

Seagate - ST4000NM0025 - 4TB, 7200RPM, 12GB/s, 128Mb cache, 512 bytes per sector

Toshiba - MG04SCA40EN - same hw specs as the seagate ( I could not associate the vdo fails with a particular hdd vendor so far - it seems to be all randomly )

When you said you mentioned there was caching, I incorrectly assumed it was additional flash drives using lvmcache. Either way, could you provide the output of sg_readcap --16 </dev/sda - j> I'm interested in knowing what the Thin Provisioning settings are. Example: sg_readcap --16 /dev/sda

The 512 bytes makes me wonder if this would be a part of the problem since vdo native is 4096, an maybe the 512 emulation should be configured.

Keep VDO at the default 4K, this woudn't be the issue.

Each server has 10 drives installed to be assigned to vms

Each server is a hypevisor node running RedHat Virtualisation ( RHHCI )

At the hypervisor level, the drives are configured as passthrough mode and given to the virtual machine.

At the moment there are six hypervisors, so 6 vms ( each vm having its own 10 drives attached ) but if I focus on one of them and have it stable, then i assume I can do the same setup for all of them.

I'm confused as to where VDO is running. IS VDO running on the hypervisor or within the VMs?

Each of the raid0 arrays are configured as per below:

Read Policy: Read Ahead Access Policy: Read Write Default Write Policy: Always Write Back Drive Cache: Enabled I/O Policy: Cached

Background Initialization: Enabled

Did this problem start occurring after the server lost power or failed in some way?

Do you see this problem when the cache is disabled and write policy is write through?

Also, can this problem be reproduced on another system?

Does the problem exist on all VDO volumes or just some?

Before creating the vdos by using cockpit UI, I have wiped each disk with "gdisk" ( maybe I should have proceed in a different way ? )

This process is fine.

All the vdos have been formated like: mkfs.xfs -K /dev/mapper/vsda

All the vdo's are mounted as per redhat documentation, by using systemd services with "discard" option

Operations

-

First setup was done with the disks attached as jbod, but I have encontered some performance issues, and thought that enabling raid mode with cache will speed up the read/writes a bit - which actually happend, but didn't solved the vdo volumes stability issues...

Yesterday I have disabled compression on all of them, and also chnaged the logical size to be 3:1 ( 10TB each volume ). After this, the number of failed vdo's reduced, but still we have failed vdo's, as per the following scenario:

After performing normal writes to the mounpoint or just a system reboot, some vdo's enter read-only mode a. some of them can be fixed by stopping and starting back with --forceRebuild option b. some of the rebuilded ones can be mounted, and some cannot be mounted due to xfs filesystem errors\

I cannot see reported hardware disk fails or errors at the server level

Below, you can find the actual confs and status and I will keep the environment as it is, for easier troubleshooting. Please let me know if any other logs are helpfull in troubleshooting this situation. Again, thank you very much for helping me in sorting this out. lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 3.7T 0 disk └─vsda 253:0 0 10T 0 vdo /hadoop01/data sdb 8:16 0 3.7T 0 disk └─vsdb 253:2 0 10T 0 vdo sdc 8:32 0 3.7T 0 disk └─vsdc 253:1 0 10T 0 vdo /hadoop03/data sdd 8:48 0 3.7T 0 disk └─vsdd 253:4 0 10T 0 vdo /hadoop04/data sde 8:64 0 3.7T 0 disk └─vsde 253:3 0 10T 0 vdo sdf 8:80 0 3.7T 0 disk └─vsdf 253:6 0 10T 0 vdo /hadoop06/data sdg 8:96 0 3.7T 0 disk └─vsdg 253:5 0 10T 0 vdo /hadoop07/data sdh 8:112 0 3.7T 0 disk └─vsdh 253:8 0 10T 0 vdo /hadoop08/data sdi 8:128 0 3.7T 0 disk └─vsdi 253:7 0 10T 0 vdo sdj 8:144 0 3.7T 0 disk └─vsdj 253:9 0 10T 0 vdo /hadoop10/data sr0 11:0 1 1024M 0 rom vda 252:0 0 70G 0 disk └─vda1 252:1 0 70G 0 part / vdostats

Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vsda 3906249728 26208420 3880041308 0% 69% /dev/mapper/vsdb 3906249728 25152932 3881096796 0% 93% /dev/mapper/vsdc 3906249728 25173216 3881076512 0% 93% /dev/mapper/vsdd 3906249728 25135564 3881114164 0% 94% /dev/mapper/vsde 3906249728 25255028 3880994700 0% 90% /dev/mapper/vsdf 3906249728 25182096 3881067632 0% 92% /dev/mapper/vsdg 3906249728 25273024 3880976704 0% 89% /dev/mapper/vsdh 3906249728 25202912 3881046816 0% 92% /dev/mapper/vsdi 3906249728 N/A N/A N/A N/A /dev/mapper/vsdj 3906249728 25339252 3880910476 0% 88% vdo status

VDO status: Date: '2019-07-26 07:10:19+00:00' Node: hdp-datanode-1.internal Kernel module: Loaded: true Name: kvdo Version information: kvdo version: 6.1.1.125 Configuration: File: /etc/vdoconf.yml Last modified: '2019-07-25 10:51:41' VDOs: vsda: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sda normal - online offline 6552105 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0445be7ab1a VDO statistics: /dev/mapper/vsda: 1K-blocks: 3906249728 1K-blocks available: 3880041308 1K-blocks used: 26208420 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 1179 bios acknowledged partial write: 0 bios acknowledged read: 1323 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 1323 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 30 bios journal discard: 0 bios journal flush: 30 bios journal fua: 30 bios journal read: 0 bios journal write: 30 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 396 bios meta completed write: 95 bios meta discard: 0 bios meta flush: 31 bios meta fua: 31 bios meta read: 396 bios meta write: 95 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 1179 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 1179 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 14 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 14 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 12 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 14 block map flush count: 0 block map found in cache: 2332 block map free pages: 32754 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 14 block map pages saved: 0 block map read count: 1834 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 313631 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 318729 flush out: 0 instance: 0 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 30 journal blocks started: 30 journal blocks writing: 0 journal blocks written: 30 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 1027410 maximum VDO IO requests in progress: 444 maximum dedupe queries: 277 no space error count: 0 operating mode: normal overhead blocks used: 6238474 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 69 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 2 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.19 write policy: sync vsdb: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdb normal - online offline 6288233 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0b504ad5ed6 VDO statistics: /dev/mapper/vsdb: 1K-blocks: 3906249728 1K-blocks available: 3881096796 1K-blocks used: 25152932 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 112 bios acknowledged partial write: 0 bios acknowledged read: 365 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 365 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 24 bios journal discard: 0 bios journal flush: 24 bios journal fua: 24 bios journal read: 0 bios journal write: 24 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 91 bios meta discard: 0 bios meta flush: 26 bios meta fua: 25 bios meta read: 260 bios meta write: 91 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 624 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 624 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1884 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 876 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 1021 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 50190 dedupe advice stale: 3 dedupe advice timeouts: 0 dedupe advice valid: 509 entries indexed: 53144 flush out: 0 instance: 2 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 24 journal blocks started: 24 journal blocks writing: 0 journal blocks written: 24 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 2042 journal entries started: 2042 journal entries writing: 0 journal entries written: 2042 logical blocks: 2684354560 logical blocks used: 768877 maximum VDO IO requests in progress: 512 maximum dedupe queries: 207 no space error count: 0 operating mode: normal overhead blocks used: 6238043 physical blocks: 976562432 posts found: 512 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 93 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 1 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 1 slabs opened: 1 slabs reopened: 1 updates found: 3 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.18 write policy: sync vsdc: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdc normal - online offline 6293304 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0e54d1017cf VDO statistics: /dev/mapper/vsdc: 1K-blocks: 3906249728 1K-blocks available: 3881076512 1K-blocks used: 25173216 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 390 bios acknowledged partial write: 0 bios acknowledged read: 643 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 643 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 10 bios journal discard: 0 bios journal flush: 10 bios journal fua: 10 bios journal read: 0 bios journal write: 10 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 75 bios meta discard: 0 bios meta flush: 11 bios meta fua: 11 bios meta read: 260 bios meta write: 75 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 390 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 390 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1653 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1154 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 55148 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 59201 flush out: 0 instance: 1 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 10 journal blocks started: 10 journal blocks writing: 0 journal blocks written: 10 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 793191 maximum VDO IO requests in progress: 516 maximum dedupe queries: 512 no space error count: 0 operating mode: normal overhead blocks used: 6238156 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 93 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdd: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdd normal - online offline 6283891 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f05e840f422d VDO statistics: /dev/mapper/vsdd: 1K-blocks: 3906249728 1K-blocks available: 3881114164 1K-blocks used: 25135564 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 356 bios acknowledged partial write: 0 bios acknowledged read: 609 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 609 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 9 bios journal discard: 0 bios journal flush: 9 bios journal fua: 9 bios journal read: 0 bios journal write: 9 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 74 bios meta discard: 0 bios meta flush: 10 bios meta fua: 10 bios meta read: 260 bios meta write: 74 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 356 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 356 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1619 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1120 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 45744 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 49805 flush out: 0 instance: 4 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 9 journal blocks started: 9 journal blocks writing: 0 journal blocks written: 9 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 765861 maximum VDO IO requests in progress: 516 maximum dedupe queries: 356 no space error count: 0 operating mode: normal overhead blocks used: 6238147 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 94 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.14 write policy: sync vsde: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sde normal - online offline 6313757 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f0191bd587b0 VDO statistics: /dev/mapper/vsde: 1K-blocks: 3906249728 1K-blocks available: 3880994700 1K-blocks used: 25255028 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 618 bios acknowledged partial write: 0 bios acknowledged read: 871 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 871 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 48 bios journal discard: 0 bios journal flush: 48 bios journal fua: 48 bios journal read: 0 bios journal write: 48 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 115 bios meta discard: 0 bios meta flush: 50 bios meta fua: 49 bios meta read: 260 bios meta write: 115 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 1130 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 1130 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 2391 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1382 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 1022 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 75649 dedupe advice stale: 2 dedupe advice timeouts: 0 dedupe advice valid: 510 entries indexed: 78489 flush out: 0 instance: 3 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 48 journal blocks started: 48 journal blocks writing: 0 journal blocks written: 48 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 2044 journal entries started: 2044 journal entries writing: 0 journal entries written: 2044 logical blocks: 2684354560 logical blocks used: 790218 maximum VDO IO requests in progress: 512 maximum dedupe queries: 210 no space error count: 0 operating mode: normal overhead blocks used: 6238108 physical blocks: 976562432 posts found: 512 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 90 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 1 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 1 slabs opened: 1 slabs reopened: 1 updates found: 2 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.22 write policy: sync vsdf: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdf normal - online offline 6295524 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f07aacef5dd4 VDO statistics: /dev/mapper/vsdf: 1K-blocks: 3906249728 1K-blocks available: 3881067632 1K-blocks used: 25182096 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 492 bios acknowledged partial write: 0 bios acknowledged read: 745 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 745 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 11 bios journal discard: 0 bios journal flush: 11 bios journal fua: 11 bios journal read: 0 bios journal write: 11 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 76 bios meta discard: 0 bios meta flush: 12 bios meta fua: 12 bios meta read: 260 bios meta write: 76 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 492 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 492 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1755 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1256 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 57396 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 61820 flush out: 0 instance: 6 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 11 journal blocks started: 11 journal blocks writing: 0 journal blocks written: 11 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 779861 maximum VDO IO requests in progress: 512 maximum dedupe queries: 175 no space error count: 0 operating mode: normal overhead blocks used: 6238128 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 92 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdg: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdg normal - online offline 6318256 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6eeacfbf3d2b0 VDO statistics: /dev/mapper/vsdg: 1K-blocks: 3906249728 1K-blocks available: 3880976704 1K-blocks used: 25273024 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 398 bios acknowledged partial write: 0 bios acknowledged read: 651 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 651 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 27 bios journal discard: 0 bios journal flush: 27 bios journal fua: 27 bios journal read: 0 bios journal write: 27 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 92 bios meta discard: 0 bios meta flush: 28 bios meta fua: 28 bios meta read: 260 bios meta write: 92 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 398 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 398 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1661 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1162 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 80074 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 84167 flush out: 0 instance: 5 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 27 journal blocks started: 27 journal blocks writing: 0 journal blocks written: 27 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 796002 maximum VDO IO requests in progress: 446 maximum dedupe queries: 297 no space error count: 0 operating mode: normal overhead blocks used: 6238182 physical blocks: 976562432 posts found: 0 posts not found: 512 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 89 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.18 write policy: sync vsdh: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdh normal - online offline 6300728 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f09cdf19554b VDO statistics: /dev/mapper/vsdh: 1K-blocks: 3906249728 1K-blocks available: 3881046816 1K-blocks used: 25202912 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 350 bios acknowledged partial write: 0 bios acknowledged read: 603 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 603 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 14 bios journal discard: 0 bios journal flush: 14 bios journal fua: 14 bios journal read: 0 bios journal write: 14 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 258 bios meta completed write: 79 bios meta discard: 0 bios meta flush: 15 bios meta fua: 15 bios meta read: 258 bios meta write: 79 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 350 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 350 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 11 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 11 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 9 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 11 block map flush count: 0 block map found in cache: 1613 block map free pages: 32757 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 11 block map pages saved: 0 block map read count: 1112 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 62608 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 63012 flush out: 0 instance: 8 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 14 journal blocks started: 14 journal blocks writing: 0 journal blocks written: 14 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 815932 maximum VDO IO requests in progress: 468 maximum dedupe queries: 0 no space error count: 0 operating mode: normal overhead blocks used: 6238120 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 92 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync vsdi: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdi read-only - online offline 6236664 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f126addf5b03 VDO statistics: /dev/mapper/vsdi: 1K-blocks: 3906249728 1K-blocks available: N/A 1K-blocks used: N/A 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 19 bios acknowledged partial write: 0 bios acknowledged read: 263 bios acknowledged write: 0 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 263 bios in write: 0 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 0 bios journal discard: 0 bios journal flush: 0 bios journal fua: 0 bios journal read: 0 bios journal write: 0 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 19 bios meta completed write: 0 bios meta discard: 0 bios meta flush: 0 bios meta fua: 0 bios meta read: 19 bios meta write: 0 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 19 bios out completed write: 0 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 19 bios out write: 0 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 3 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 3 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 2 block map dirty pages: 0 block map discard required: 0 block map failed pages: 0 block map failed reads: 1 block map failed writes: 0 block map fetch required: 3 block map flush count: 0 block map found in cache: 259 block map free pages: 32766 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 3 block map pages saved: 0 block map read count: 262 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 0 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: N/A dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 72140 flush out: 0 instance: 7 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 0 journal blocks started: 0 journal blocks writing: 0 journal blocks written: 0 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 0 journal entries started: 0 journal entries writing: 0 journal entries written: 0 logical blocks: 2684354560 logical blocks used: 852316 maximum VDO IO requests in progress: 30 maximum dedupe queries: 0 no space error count: 0 operating mode: read-only overhead blocks used: 6238222 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: N/A slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 0 slabs reopened: 0 updates found: 0 updates not found: 0 used percent: N/A version: 28 write amplification ratio: 0.0 write policy: sync vsdj: Acknowledgement threads: 1 Activate: enabled Bio rotation interval: 64 Bio submission threads: 4 Block map cache size: 128M Block map period: 16380 Block size: 4096 CPU-work threads: 2 Compression: disabled Configured write policy: auto Deduplication: enabled Device mapper status: 0 21474836480 vdo /dev/sdj normal - online offline 6334813 976562432 Emulate 512 byte: disabled Hash zone threads: 1 Index checkpoint frequency: 0 Index memory setting: 2 Index parallel factor: 0 Index sparse: disabled Index status: online Logical size: 10T Logical threads: 1 Physical size: 3814697M Physical threads: 1 Read cache: disabled Read cache size: 0M Slab size: 2G Storage device: /dev/disk/by-id/scsi-3600062b20314aa0024c6f02e3c597e3b VDO statistics: /dev/mapper/vsdj: 1K-blocks: 3906249728 1K-blocks available: 3880910476 1K-blocks used: 25339252 512 byte emulation: false KVDO module bios used: 373084 KVDO module bytes used: 30381531648 KVDO module peak bio count: 373372 KVDO module peak bytes used: 30381593936 bios acknowledged discard: 0 bios acknowledged flush: 0 bios acknowledged fua: 0 bios acknowledged partial discard: 0 bios acknowledged partial flush: 0 bios acknowledged partial fua: 0 bios acknowledged partial read: 354 bios acknowledged partial write: 0 bios acknowledged read: 607 bios acknowledged write: 512 bios in discard: 0 bios in flush: 0 bios in fua: 0 bios in partial discard: 0 bios in partial flush: 0 bios in partial fua: 0 bios in partial read: 0 bios in partial write: 0 bios in progress discard: 0 bios in progress flush: 0 bios in progress fua: 0 bios in progress read: 0 bios in progress write: 0 bios in read: 607 bios in write: 512 bios journal completed discard: 0 bios journal completed flush: 0 bios journal completed fua: 0 bios journal completed read: 0 bios journal completed write: 11 bios journal discard: 0 bios journal flush: 11 bios journal fua: 11 bios journal read: 0 bios journal write: 11 bios meta completed discard: 0 bios meta completed flush: 0 bios meta completed fua: 0 bios meta completed read: 260 bios meta completed write: 76 bios meta discard: 0 bios meta flush: 12 bios meta fua: 12 bios meta read: 260 bios meta write: 76 bios out completed discard: 0 bios out completed flush: 0 bios out completed fua: 0 bios out completed read: 354 bios out completed write: 512 bios out discard: 0 bios out flush: 0 bios out fua: 0 bios out read: 354 bios out write: 512 bios page cache completed discard: 0 bios page cache completed flush: 0 bios page cache completed fua: 0 bios page cache completed read: 13 bios page cache completed write: 0 bios page cache discard: 0 bios page cache flush: 0 bios page cache fua: 0 bios page cache read: 13 bios page cache write: 0 block map cache pressure: 0 block map cache size: 134217728 block map clean pages: 11 block map dirty pages: 2 block map discard required: 0 block map failed pages: 0 block map failed reads: 0 block map failed writes: 0 block map fetch required: 13 block map flush count: 0 block map found in cache: 1617 block map free pages: 32755 block map incoming pages: 0 block map outgoing pages: 0 block map pages loaded: 13 block map pages saved: 0 block map read count: 1118 block map read outgoing: 0 block map reclaimed: 0 block map wait for page: 0 block map write count: 512 block size: 4096 completed recovery count: 0 compressed blocks written: 0 compressed fragments in packer: 0 compressed fragments written: 0 concurrent data matches: 0 concurrent hash collisions: 0 current VDO IO requests in progress: 0 current dedupe queries: 0 data blocks used: 96576 dedupe advice stale: 0 dedupe advice timeouts: 0 dedupe advice valid: 0 entries indexed: 96941 flush out: 0 instance: 9 invalid advice PBN count: 0 journal blocks batching: 0 journal blocks committed: 11 journal blocks started: 11 journal blocks writing: 0 journal blocks written: 11 journal commits requested count: 0 journal disk full count: 0 journal entries batching: 0 journal entries committed: 1024 journal entries started: 1024 journal entries writing: 0 journal entries written: 1024 logical blocks: 2684354560 logical blocks used: 846871 maximum VDO IO requests in progress: 512 maximum dedupe queries: 0 no space error count: 0 operating mode: normal overhead blocks used: 6238237 physical blocks: 976562432 posts found: 0 posts not found: 0 queries found: 0 queries not found: 0 read cache accesses: 0 read cache data hits: 0 read cache hits: 0 read only error count: 0 read-only recovery count: 0 recovery progress (%): N/A reference blocks written: 0 release version: 131337 saving percent: 88 slab count: 1852 slab journal blocked count: 0 slab journal blocks written: 0 slab journal disk full count: 0 slab journal flush count: 0 slab journal tail busy count: 0 slab summary blocks written: 0 slabs opened: 1 slabs reopened: 1 updates found: 0 updates not found: 0 used percent: 0 version: 28 write amplification ratio: 1.15 write policy: sync Some of the latest logs cat /var/log/messages | grep "kernel: XFS"

Jul 26 12:25:28 hdp-datanode-1 kernel: XFS (vda1): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-1): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-0): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-6): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-8): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-9): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-5): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): metadata I/O error: block 0x280000010 ("xlog_bread_noalign") error 5 numblks 8 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): empty log check failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): log mount/recovery failed: error -5 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-7): log mount failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-4): Mounting V5 Filesystem Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-5): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): Corruption warning: Metadata has LSN (1:2736) ahead of current LSN (1:120). Please unmount and run xfs_repair (>= v4.3) to resolve. Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): log mount/recovery failed: error -22 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-2): log mount failed Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-1): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-4): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-6): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-8): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-0): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): Corruption warning: Metadata has LSN (1:4824) ahead of current LSN (1:1984). Please unmount and run xfs_repair (>= v4.3) to resolve. Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-9): Ending clean mount Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): log mount/recovery failed: error -22 Jul 26 05:55:54 hdp-datanode-1 kernel: XFS (dm-3): log mount failed cat /var/log/messages | grep vdo

Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:journalQ: VDO commencing normal operation Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: Jul 26 05:55:38 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: Jul 26 05:55:38 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: device 'vsdb' started Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: resuming device 'vsdb' Jul 26 05:55:38 hdp-datanode-1 kernel: kvdo2:dmsetup: device 'vsdb' resumed Jul 26 05:55:39 hdp-datanode-1 UDS/vdodmeventd[3694]: INFO (vdodmeventd/3694) VDO device vsdb is now registered with dmeventd for monitoring Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: starting device 'vsde' Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: Using write policy sync automatically. Jul 26 05:55:39 hdp-datanode-1 kernel: kvdo3:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:39 hdp-datanode-1 kernel: uds: kvdo0:dedupeQ: read index page map, last update 0 Jul 26 05:55:39 hdp-datanode-1 kernel: uds: kvdo0:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:journalQ: VDO commencing normal operation Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: device 'vsde' started Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: resuming device 'vsde' Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo3:dmsetup: device 'vsde' resumed Jul 26 05:55:41 hdp-datanode-1 UDS/vdodmeventd[3790]: INFO (vdodmeventd/3790) VDO device vsde is now registered with dmeventd for monitoring Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: starting device 'vsdd' Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: Using write policy sync automatically. Jul 26 05:55:41 hdp-datanode-1 kernel: kvdo4:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo1:dedupeQ: read index page map, last update 0 Jul 26 05:55:41 hdp-datanode-1 kernel: uds: kvdo1:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:42 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: read index page map, last update 0 Jul 26 05:55:42 hdp-datanode-1 kernel: uds: kvdo2:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:journalQ: VDO commencing normal operation Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: Jul 26 05:55:43 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: Jul 26 05:55:43 hdp-datanode-1 kernel: uds: kvdo4:dedupeQ: Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: device 'vsdd' started Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: resuming device 'vsdd' Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo4:dmsetup: device 'vsdd' resumed Jul 26 05:55:43 hdp-datanode-1 UDS/vdodmeventd[3926]: INFO (vdodmeventd/3926) VDO device vsdd is now registered with dmeventd for monitoring Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: starting device 'vsdg' Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: Using write policy sync automatically. Jul 26 05:55:43 hdp-datanode-1 kernel: kvdo5:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5 Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: read index page map, last update 0 Jul 26 05:55:45 hdp-datanode-1 kernel: uds: kvdo3:dedupeQ: index_0: loaded index from chapter 0 through chapter 0 Jul 26

skunkr commented 5 years ago

Thank you so much for the input, and im also sorry for being so late. At this moment we have reconfigured all the passthrough disks and have disable vdo, just to test and maybe isolate the issue. After this, i will revert to the vdo based environment and do the tests again. I will come back soon with new infos, since we really count on vdo technology to work. Thing that im sure will be possible with the correct setup. Thank you very much !

skunkr commented 5 years ago

Comming back to vdo configuration today, the jbod disks passthrough to vms tests went fine so far. The only thing is that I had to add the physical disks to vms as "direct luns" instead of regular passthrough. Before going to reconfigure everything, it is a bit confusing to me what is the best approach to do it (inside the vm):

create the vdo directly on the device
create a partition on the device and after that create the vdo ontop of the partition
create a lvm on the device and then create vdo ontop of the lvm After any of these cases:
- create a partition on the vdo && format it with mkfs.xfs -K && mount OR
- directly format the vdo with mkfs and mount without creating a partition Could you please let me know what would it be a recomended procedure ? Thank you so much !

Leo

rhawalsh commented 5 years ago

Thanks for getting back to us @skunkr.

It is always best to place VDO on top of a device that can be extended, whether it's a raw partition or an LVM volume or some other format that allows extension shouldn't matter all that much. You may be able to realize better performance out of one type versus the other, but that depends on your needs. If you're trying to squeeze every last bit of performance you can out of the device, you'll need to do some profiling. You can use the Tuning VDO section of the VDO Integration Guide to get started with that process.

To specifically answer your questions: Inside a VM configuration, I would probably personally opt for using LVM underneath, that way you can simply add additional storage devices, add them to the VG, extend the underlying LV servicing VDO, and finally extending the VDO volume via vdo growPhysical.

To answer your second question, it depends on the use case. If you're just simply intending to use the VDO device for a single mountpoint (as opposed to carving up the VDO device into multiple volumes), then formatting the VDO device directly and then mounting it should be sufficient.

skunkr commented 5 years ago

Thank you so much for your answer, it really helps ! It seems so far that vdo devices are stable, and the problem seemed to be at the passthrough level of the hypervisor. But of course just for not having all the things working seemless, we have a dramatical write performance decrease... To be a bit more specific,:

having the disk mounted we had like 600mb/s reads and 115mb/s writes
using the disk as vdo ( vdo created xfs formated and mounted ) gives 1200mb reads and only 16mb/s writes
the tests where done by using hadoop native dfsio tests, but if needed, i can come back with some fio tests on the drives. I have read the RedHat article regarding tuning, but it seems a bit confusing, there are a lot of parameters that can be tweaked, and i'm just not sure what performance setting should I apply to increase the writes at least at the amount of what we previously had... Could you please recommend what would be some appropiate settings on VDO / OS side being that the vms have planty of resources ( 128GB ram, 12 vCpu's ) ? I would really appreciate any help. Thank you so much again !

Leo

rhawalsh commented 4 years ago

Hi @skunkr,

Apologies for not getting back within a reasonable amount of time. Tuning VDO is highly system and workload dependent. Working through the VDO Evaluation Guide section of Performance Testing might help.

I would suggest if you have specific questions, then we can try to help out. Otherwise it's not a simple answer of workload X works best with a specific set of parameters without having to do some testing first.

dm-vdo / kvdo

Errors and random vdo volume failures #25

vdostats

cat /var/log/messages | grep vdo

vdo status -n vsdh

HW and Setup Infos

Operations

lsblk

vdostats

vdo status

Some of the latest logs

cat /var/log/messages | grep "kernel: XFS"

cat /var/log/messages | grep vdo

First setup was done with the disks attached as jbod, but I have encontered some performance issues, and thought that enabling raid mode with cache will speed up the read/writes a bit - which actually happend, but didn't solved the vdo volumes stability issues...

Yesterday I have disabled compression on all of them, and also chnaged the logical size to be 3:1 ( 10TB each volume ). After this, the number of failed vdo's reduced, but still we have failed vdo's, as per the following scenario:

After performing normal writes to the mounpoint or just a system reboot, some vdo's enter read-only mode a. some of them can be fixed by stopping and starting back with --forceRebuild option b. some of the rebuilded ones can be mounted, and some cannot be mounted due to xfs filesystem errors\