Open jhindersson opened 1 year ago
KlaudiaJ
commented
1 year ago Hi @jhindersson, let me ask you a few more questions.
jhindersson
commented
1 year ago Hi,
Absolutely!
# casadm -P -i 1 -d 1
IO class ID 1
IO class name metadata&done
Eviction priority 0
Max size 100%╔══════════════════╤═══════════╤═══════╤═════════════╗ ║ Usage statistics │ Count │ % │ Units ║ ╠══════════════════╪═══════════╪═══════╪═════════════╣ ║ Occupancy │ 116019352 │ 13.1 │ 4KiB Blocks ║ ║ Clean │ 116016848 │ 100.0 │ 4KiB Blocks ║ ║ Dirty │ 2504 │ 0.0 │ 4KiB Blocks ║ ╚══════════════════╧═══════════╧═══════╧═════════════╝
╔══════════════════════╤════════════╤═══════╤══════════╗ ║ Request statistics │ Count │ % │ Units ║ ╠══════════════════════╪════════════╪═══════╪══════════╣ ║ Read hits │ 551969263 │ 50.7 │ Requests ║ ║ Read partial misses │ 0 │ 0.0 │ Requests ║ ║ Read full misses │ 4679553 │ 0.4 │ Requests ║ ║ Read total │ 556648816 │ 51.2 │ Requests ║ ╟──────────────────────┼────────────┼───────┼──────────╢ ║ Write hits │ 183552476 │ 16.9 │ Requests ║ ║ Write partial misses │ 0 │ 0.0 │ Requests ║ ║ Write full misses │ 945126 │ 0.1 │ Requests ║ ║ Write total │ 184497602 │ 17.0 │ Requests ║ ╟──────────────────────┼────────────┼───────┼──────────╢ ║ Pass-Through reads │ 80155392 │ 7.4 │ Requests ║ ║ Pass-Through writes │ 266631853 │ 24.5 │ Requests ║ ║ Serviced requests │ 741146418 │ 68.1 │ Requests ║ ╟──────────────────────┼────────────┼───────┼──────────╢ ║ Total requests │ 1087933663 │ 100.0 │ Requests ║ ╚══════════════════════╧════════════╧═══════╧══════════╝
╔══════════════════════════════════╤════════════╤═══════╤═════════════╗ ║ Block statistics │ Count │ % │ Units ║ ╠══════════════════════════════════╪════════════╪═══════╪═════════════╣ ║ Reads from core(s) │ 84836276 │ 7.6 │ 4KiB Blocks ║ ║ Writes to core(s) │ 1034347789 │ 92.4 │ 4KiB Blocks ║ ║ Total to/from core(s) │ 1119184065 │ 100.0 │ 4KiB Blocks ║ ╟──────────────────────────────────┼────────────┼───────┼─────────────╢ ║ Reads from cache │ 1503276395 │ 88.8 │ 4KiB Blocks ║ ║ Writes to cache │ 189177155 │ 11.2 │ 4KiB Blocks ║ ║ Total to/from cache │ 1692453550 │ 100.0 │ 4KiB Blocks ║ ╟──────────────────────────────────┼────────────┼───────┼─────────────╢ ║ Reads from exported object(s) │ 636806806 │ 56.9 │ 4KiB Blocks ║ ║ Writes to exported object(s) │ 481529534 │ 43.1 │ 4KiB Blocks ║ ║ Total to/from exported object(s) │ 1118336340 │ 100.0 │ 4KiB Blocks ║ ╚══════════════════════════════════╧════════════╧═══════╧═════════════╝
2. I would classify it as quite static at around 100 IOPS per second for writes. Sure there are a few shorter increases but still within 10-20% of "normal" amount. For reads it is quite variable but not really bursty. It is very hard to quantify but we are talking in the range of 500-2000 IOPS which we have tested for well above 400K IOPS.
3. Yes, i would say that they are steadily increasing at a rate of around 200 per second.
I do have a feeling that it might be related to another (unverified) feeling that i have, which is that clean data in io-classes with higher eviction priority is not evicted to make place for new data in lower eviction priority classes, that is writes and reads to metadata does not evict large files from cache even though it is clean and have a high eviction priority, it is however nothing that I have taken the time to verify yet.
Anything else i can contribute to help troubleshoot?Thank you @jhindersson for more information!
There is one more test that you could run to narrow down the possible root causes. That is to run the workload with ALRU cleaning policy with idle time set to '0'.
jhindersson
commented
1 year ago Yes, i already have one node running ALRU with idle time set to 2 just for testing as i read older issues where ACP might cause these kind of issues however it does not seem to do any difference in our case.
Changing to ALRU with idle time 0 did not make any difference either, cache is 100% cleaned just as with ACP but pass-through reads/writes still rises steadily.
What are the possible reasons for hitting the pass-through path?
# casadm -G -n cleaning -i 1
╔══════════════════════╤═══════╗
║ Parameter name │ Value ║
╠══════════════════════╪═══════╣
║ Cleaning policy type │ alru ║
╚══════════════════════╧═══════
# casadm -G -n cleaning-alru -i 1
╔═════════════════════════╤═══════╗
║ Parameter name │ Value ║
╠═════════════════════════╪═══════╣
║ Wake up time [s] │ 20 ║
║ Stale buffer time [s] │ 120 ║
║ Flush max buffers │ 100 ║
║ Activity threshold [ms] │ 0 ║
╚═════════════════════════╧═══════╝
jfckm
commented
1 year ago Hi Joakim,
Thanks a lot for filling an issue and for in-depth investigation! It looks like it might be difficult for us to replicate what you have exactly, but we're thinking about developing some bpftrace script to analyze potential issues which might be triggering P/T in your case. Would you be OK with running that and sharing results with us? I think we'd take until next week to do that as we're quite swamped with work right now.
Let me know what you think!
jhindersson
commented
1 year ago Sure, i can run that if you think it can help figuring out what is causing this.
Take your time, there is no urgency in this.
Thank you for helping out!
jfckm
commented
1 year ago Hi again,
Sorry for the long wait! Unfortunately you'll have to wait some more - I tried setting up some sane debugging platform for Ubuntu 18.04 \w HWE but it doesn't cooperate with BPF tools. We'll do some code-review and maybe some experiments and I'll let you know what came of it next week.
Thanks for your patience.
mmichal10
commented
1 year ago Hello @jhindersson
could you please share with us output of cat /sys/module/cas_cache/parameters/unaligned_io and iostat -xmt 30 20 | tee iostat.out with the workload running in background?
jhindersson
commented
1 year ago Hi,
Sure, the unaligned_io is set to 1 as per module defaults. Which should not pass-through requests even if we miss alignment if i understand correctly?
iostat output, this is during an active rebalance so a littler higher load than usual but you can see some very high write latency (seconds) to cache devices compared to the core devices:
Linux 5.4.0-104-generic (<hostname>) 12/06/22 _x86_64_ (16 CPU)
12/06/22 11:47:33
avg-cpu: %user %nice %system %iowait %steal %idle
36.14 0.00 10.10 5.09 0.00 48.67
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 1.00 0.00
nvme0n1 483.09 1072.61 8.28 4.51 358.62 14.34 42.61 1.32 0.17 0.04 0.01 17.55 4.31 0.10 16.16
nvme1n1 2.13 46.43 0.10 0.40 0.13 3.89 5.72 7.72 0.19 0.26 0.01 49.50 8.72 0.16 0.79
dm-0 2.23 50.24 0.10 0.40 0.00 0.00 0.00 0.00 0.22 0.65 0.03 46.26 8.07 0.15 0.79
sdc 20.34 16.10 0.74 0.87 0.02 0.09 0.09 0.54 0.76 13.14 0.18 37.10 55.16 4.49 16.37
sdg 22.38 16.11 0.95 0.87 0.04 0.09 0.18 0.57 2.80 1.60 0.04 43.42 55.37 4.54 17.46
sdf 18.52 16.15 0.87 0.87 0.03 0.09 0.16 0.55 11.46 13.03 0.17 48.16 55.17 4.47 15.50
sdb 21.26 17.30 0.80 1.01 0.01 0.17 0.05 0.97 0.79 6.31 0.08 38.47 59.54 4.36 16.80
sdi 18.31 15.92 0.54 0.89 0.03 0.09 0.14 0.58 11.52 13.06 0.16 30.18 56.97 4.31 14.75
sdj 17.28 16.20 0.72 0.88 0.02 0.09 0.12 0.56 0.08 0.60 0.19 42.51 55.38 4.57 15.30
sdh 20.19 15.77 0.94 0.85 0.00 0.08 0.02 0.53 9.87 12.37 0.14 47.60 55.27 4.22 15.19
sde 19.53 16.19 0.72 0.88 0.02 0.09 0.11 0.58 1.53 0.54 0.21 37.79 55.66 4.67 16.68
sdk 20.64 16.13 0.79 0.89 0.02 0.09 0.08 0.55 1.88 0.61 0.00 39.19 56.40 4.56 16.78
sdl 20.01 16.22 0.80 0.88 0.07 0.09 0.35 0.56 10.65 12.85 0.16 40.81 55.67 4.35 15.77
sda 20.80 15.80 0.88 0.86 0.04 0.09 0.17 0.54 0.30 12.56 0.16 43.09 55.75 4.25 15.56
sdd 17.28 16.16 0.75 0.87 0.04 0.09 0.22 0.54 11.82 0.82 0.18 44.35 55.30 4.39 14.66
cas1-1 64.92 38.45 1.06 0.68 0.00 0.00 0.00 0.00 0.61 2.32 0.13 16.70 18.12 1.06 10.95
cas1-2 67.83 38.89 0.97 0.69 0.00 0.00 0.00 0.00 3.20 5.51 0.21 14.57 18.24 1.08 11.53
cas1-3 68.27 38.93 1.06 0.68 0.00 0.00 0.00 0.00 0.54 0.16 0.04 15.88 17.98 1.14 12.20
cas1-4 68.70 39.11 0.96 0.68 0.00 0.00 0.00 0.00 0.33 4.75 0.21 14.38 17.92 0.97 10.44
cas1-5 68.12 39.15 0.96 0.70 0.00 0.00 0.00 0.00 0.62 5.51 0.04 14.46 18.32 1.13 12.16
cas1-6 69.01 39.06 1.07 0.69 0.00 0.00 0.00 0.00 0.45 1.23 0.08 15.83 18.12 1.03 11.11
cas1-7 70.90 39.08 1.13 0.69 0.00 0.00 0.00 0.00 1.56 1.54 0.17 16.38 18.20 1.19 13.13
cas1-8 67.09 38.40 1.12 0.68 0.00 0.00 0.00 0.00 3.06 0.11 0.21 17.14 18.06 1.03 10.84
cas1-9 64.17 38.41 1.04 0.71 0.00 0.00 0.00 0.00 0.23 1.27 0.06 16.57 18.80 1.05 10.78
cas1-10 68.50 39.17 0.96 0.69 0.00 0.00 0.00 0.00 0.27 1.69 0.09 14.35 18.14 1.01 10.87
cas1-11 67.71 39.15 0.96 0.70 0.00 0.00 0.00 0.00 0.83 4.06 0.22 14.57 18.22 1.13 12.08
cas1-12 66.06 39.06 0.97 0.69 0.00 0.00 0.00 0.00 0.30 4.26 0.19 15.09 18.21 1.04 10.95
12/06/22 11:48:03
avg-cpu: %user %nice %system %iowait %steal %idle
34.38 0.00 10.39 32.44 0.00 22.79
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 881.47 1377.37 16.05 5.46 1509.53 16.03 63.13 1.15 0.21 0.03 0.00 18.65 4.06 0.25 57.15
nvme1n1 0.27 45.83 0.01 0.50 1.00 3.77 78.95 7.59 2.12 0.25 0.00 31.50 11.11 0.14 0.63
dm-0 1.23 49.50 0.01 0.50 0.00 0.00 0.00 0.00 0.43 0.52 0.03 6.38 10.31 0.13 0.64
sdc 59.87 11.27 0.71 0.30 0.00 0.00 0.00 0.00 16.51 1.77 0.89 12.10 27.07 3.63 25.84
sdg 173.53 9.43 8.71 0.27 0.00 0.00 0.00 0.00 10.99 3.62 1.69 51.41 29.40 4.42 80.87
sdf 60.77 52.53 0.61 20.26 0.00 0.00 0.00 0.00 18.65 65.52 4.36 10.28 394.87 2.95 33.47
sdb 394.27 7.43 1.89 0.24 0.00 0.00 0.00 0.00 15.72 5.22 5.50 4.91 32.97 1.58 63.43
sdi 46.97 12.03 0.52 0.34 0.00 0.00 0.00 0.00 19.81 2.66 0.86 11.40 28.53 3.98 23.48
sdj 47.53 11.77 0.68 0.30 0.00 0.00 0.00 0.00 17.79 5.25 0.80 14.60 25.73 3.96 23.51
sdh 42.43 8.50 0.50 0.21 0.00 0.00 0.00 0.00 13.52 4.88 0.53 12.12 25.38 4.09 20.85
sde 63.37 11.27 0.64 0.29 0.00 0.00 0.00 0.00 14.11 1.17 0.77 10.27 26.09 3.57 26.64
sdk 174.13 12.60 0.95 0.33 0.00 0.00 0.00 0.00 40.74 2.30 6.79 5.59 26.69 4.73 88.40
sdl 628.70 154.10 3.00 5.92 0.00 0.00 0.00 0.00 45.43 33.34 32.23 4.88 39.31 0.94 73.40
sda 208.57 16.60 1.29 5.09 0.00 0.00 0.00 0.00 6.38 37.55 1.75 6.31 314.02 2.89 65.08
sdd 48.37 8.43 0.54 0.27 0.00 0.00 0.00 0.00 15.81 2.60 0.68 11.46 33.01 3.47 19.73
cas1-1 378.37 21.10 2.03 4.99 0.00 0.00 0.00 0.00 4.21 191.13 5.62 5.50 242.24 1.62 64.67
cas1-2 517.13 21.73 2.59 0.16 0.00 0.00 0.00 0.00 18.65 1.39 9.68 5.14 7.60 1.22 65.68
cas1-3 189.23 30.20 1.23 0.16 0.00 0.00 0.00 0.00 21.05 0.45 4.00 6.67 5.40 1.23 26.91
cas1-4 161.03 24.67 1.18 0.19 0.00 0.00 0.00 0.00 5.07 2.46 0.88 7.50 8.06 1.08 20.03
cas1-5 165.27 27.57 1.08 0.15 0.00 0.00 0.00 0.00 5.90 0.20 0.98 6.71 5.45 1.20 23.12
cas1-6 176.40 51.00 1.14 20.12 0.00 0.00 0.00 0.00 7.00 60.28 4.31 6.60 404.05 1.41 31.97
cas1-7 827.83 23.40 11.32 0.14 0.00 0.00 0.00 0.00 2.87 1.62 2.41 14.00 6.31 0.97 82.63
cas1-8 139.30 22.10 0.95 0.12 0.00 0.00 0.00 0.00 5.15 2.10 0.76 6.96 5.47 1.15 18.60
cas1-9 186.13 32.70 1.18 0.20 0.00 0.00 0.00 0.00 11.92 1.82 2.28 6.50 6.12 1.06 23.25
cas1-10 169.83 28.47 1.18 0.15 0.00 0.00 0.00 0.00 4.85 0.55 0.84 7.13 5.50 1.05 20.80
cas1-11 248.37 35.83 1.26 0.19 0.00 0.00 0.00 0.00 31.25 1.21 7.80 5.21 5.38 3.16 89.72
cas1-12 769.70 324.57 3.59 4.31 0.00 0.00 0.00 0.00 42.25 212.45 101.48 4.77 13.58 0.67 73.43
12/06/22 11:48:33
avg-cpu: %user %nice %system %iowait %steal %idle
30.76 0.00 9.46 29.20 0.00 30.58
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 528.77 860.30 8.74 3.48 982.23 26.37 65.01 2.97 0.20 0.04 0.00 16.93 4.14 0.31 42.60
nvme1n1 0.30 30.83 0.00 0.33 0.00 3.77 0.00 10.89 6.11 0.31 0.01 11.56 10.85 0.20 0.63
dm-0 0.30 34.57 0.00 0.33 0.00 0.00 0.00 0.00 6.22 1.13 0.04 11.56 9.70 0.18 0.63
sdc 49.30 8.00 0.49 0.19 0.00 0.00 0.00 0.00 25.46 1.75 1.16 10.16 24.35 2.88 16.52
sdg 90.50 7.93 3.10 0.20 0.00 0.00 0.00 0.00 19.68 3.16 1.66 35.11 25.78 3.84 37.77
sdf 76.67 5.20 1.03 0.14 0.00 0.00 0.00 0.00 9.45 1.89 0.62 13.69 27.13 2.92 23.91
sdb 335.57 9.47 1.53 0.25 0.00 0.00 0.00 0.00 14.72 10.45 4.43 4.66 27.25 1.49 51.52
sdi 36.57 3.53 0.31 0.08 0.00 0.00 0.00 0.00 11.29 1.36 0.35 8.58 22.98 3.11 12.48
sdj 22.17 2.77 0.42 0.07 0.00 0.00 0.00 0.00 13.62 1.04 0.26 19.55 25.73 4.60 11.48
sdh 39.40 5.37 0.50 0.16 0.00 0.00 0.00 0.00 12.63 2.29 0.43 12.90 30.58 3.20 14.31
sde 38.87 6.03 0.32 0.16 0.00 0.00 0.00 0.00 16.85 1.38 0.58 8.49 26.56 3.36 15.08
sdk 186.40 11.67 0.90 0.27 0.00 0.00 0.00 0.00 34.48 4.46 6.12 4.94 24.08 3.16 62.53
sdl 37.70 156.37 0.27 6.40 0.00 0.00 0.00 0.00 37.08 183.27 29.73 7.40 41.92 2.56 49.60
sda 105.70 9.87 0.73 0.23 0.00 0.00 0.00 0.00 8.57 4.50 0.84 7.07 23.46 2.79 32.19
sdd 43.27 9.13 0.65 0.27 0.00 0.00 0.00 0.00 18.44 1.90 0.72 15.45 30.79 3.36 17.60
cas1-1 196.87 22.83 1.17 0.12 0.00 0.00 0.00 0.00 7.62 6.69 1.65 6.08 5.26 2.10 46.07
cas1-2 436.13 24.57 1.97 0.13 0.00 0.00 0.00 0.00 41.38 3.01 18.12 4.62 5.48 1.45 66.96
cas1-3 145.37 18.13 0.88 0.11 0.00 0.00 0.00 0.00 10.62 13.65 1.79 6.21 6.29 1.26 20.59
cas1-4 138.37 27.20 1.04 0.20 0.00 0.00 0.00 0.00 8.16 0.61 1.15 7.68 7.44 2.00 33.09
cas1-5 125.23 15.53 0.67 0.09 0.00 0.00 0.00 0.00 25.74 0.58 3.23 5.49 5.61 1.96 27.56
cas1-6 168.53 22.17 1.40 0.15 0.00 0.00 0.00 0.00 11.36 29.37 2.57 8.51 6.78 2.01 38.32
cas1-7 352.13 19.60 4.14 0.10 0.00 0.00 0.00 0.00 7.83 2.81 2.81 12.04 5.15 1.50 55.85
cas1-8 127.60 19.53 0.85 0.11 0.00 0.00 0.00 0.00 4.21 5.14 0.64 6.82 5.95 1.16 17.11
cas1-9 112.77 16.77 0.64 0.09 0.00 0.00 0.00 0.00 35.07 17.53 4.25 5.78 5.33 1.43 18.57
cas1-10 126.33 10.63 0.88 0.06 0.00 0.00 0.00 0.00 8.08 0.76 1.03 7.11 5.97 2.17 29.71
cas1-11 268.17 27.57 1.24 0.15 0.00 0.00 0.00 0.00 29.95 45.15 9.27 4.72 5.56 2.46 72.65
cas1-12 136.80 307.77 0.68 6.88 0.00 0.00 0.00 0.00 11.05 1924.17 593.71 5.06 22.90 1.23 54.77
12/06/22 11:49:03
avg-cpu: %user %nice %system %iowait %steal %idle
35.72 0.00 10.51 30.23 0.00 23.54
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1283.80 1367.67 22.47 5.48 2783.90 27.87 68.44 2.00 0.20 0.03 0.00 17.92 4.10 0.22 59.53
nvme1n1 0.10 49.93 0.00 0.56 0.00 3.93 0.00 7.30 5.33 0.26 0.01 4.00 11.39 0.13 0.67
dm-0 0.10 53.73 0.00 0.56 0.00 0.00 0.00 0.00 5.33 0.59 0.03 4.00 10.60 0.12 0.67
sdc 59.67 14.37 0.50 0.37 0.00 0.00 0.00 0.00 20.00 1.80 1.09 8.57 26.65 3.74 27.69
sdg 162.77 8.47 9.26 0.37 0.00 0.00 0.00 0.00 13.74 4.75 2.05 58.24 44.54 4.17 71.32
sdf 203.40 21.30 1.04 0.46 0.00 0.00 0.00 0.00 6.72 8.03 1.35 5.24 22.31 1.93 43.29
sdb 287.70 7.03 1.60 0.20 0.00 0.00 0.00 0.00 6.64 2.33 1.55 5.68 28.55 1.17 34.55
sdi 46.63 10.40 0.51 0.30 0.00 0.00 0.00 0.00 12.68 1.53 0.50 11.19 29.68 4.33 24.68
sdj 78.13 9.37 0.65 0.31 0.00 0.00 0.00 0.00 15.16 6.75 1.11 8.58 33.41 4.80 42.01
sdh 147.87 13.60 0.83 0.84 0.00 0.00 0.00 0.00 8.42 13.11 1.29 5.76 62.91 1.69 27.33
sde 50.33 6.20 0.51 0.17 0.00 0.00 0.00 0.00 13.66 3.11 0.61 10.38 28.15 3.87 21.85
sdk 332.43 17.37 1.75 2.57 0.00 0.00 0.00 0.00 28.97 58.67 10.11 5.38 151.27 2.64 92.41
sdl 107.57 160.00 0.86 5.64 0.00 0.67 0.00 0.41 54.93 25.49 9.69 8.15 36.08 2.48 66.37
sda 50.07 10.90 0.57 0.29 0.00 0.00 0.00 0.00 15.57 3.95 0.72 11.65 27.46 4.15 25.31
sdd 462.57 38.83 3.12 0.90 0.00 0.00 0.00 0.00 10.07 14.01 4.70 6.90 23.70 1.33 66.84
cas1-1 183.40 39.40 1.12 0.25 0.00 0.00 0.00 0.00 5.32 3.15 1.10 6.27 6.42 1.20 26.80
cas1-2 436.33 24.27 2.22 0.14 0.00 0.00 0.00 0.00 5.19 0.42 2.27 5.22 5.81 0.84 38.77
cas1-3 178.40 42.37 0.99 0.23 0.00 0.00 0.00 0.00 7.51 56.55 3.74 5.71 5.49 1.14 25.09
cas1-4 2221.93 145.90 10.02 1.63 0.00 0.00 0.00 0.00 2.58 24.31 9.28 4.62 11.47 0.29 68.68
cas1-5 181.27 31.40 1.05 0.18 0.00 0.00 0.00 0.00 14.56 1.86 2.70 5.92 5.86 1.18 25.07
cas1-6 331.07 52.67 1.58 0.28 0.00 0.00 0.00 0.00 5.69 2.14 2.00 4.88 5.43 1.06 40.68
cas1-7 560.97 26.17 11.62 0.29 0.00 0.00 0.00 0.00 6.05 1.05 3.42 21.21 11.46 1.27 74.44
cas1-8 256.43 36.60 1.27 0.74 0.00 0.00 0.00 0.00 6.12 3.32 1.69 5.08 20.83 1.28 37.43
cas1-9 179.50 23.43 1.06 0.15 0.00 0.00 0.00 0.00 6.57 9.57 1.40 6.05 6.59 1.21 24.57
cas1-10 381.20 21.60 1.87 0.19 0.00 0.00 0.00 0.00 4.69 0.61 1.80 5.02 8.82 1.01 40.77
cas1-11 473.00 46.67 2.32 2.47 0.00 0.00 0.00 0.00 37.63 32.14 19.30 5.03 54.14 1.82 94.59
cas1-12 258.77 218.10 1.51 4.54 0.00 0.00 0.00 0.00 20.13 687.96 155.25 5.96 21.32 1.29 61.48
12/06/22 11:49:33
avg-cpu: %user %nice %system %iowait %steal %idle
32.61 0.00 10.11 31.60 0.00 25.68
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 901.43 1781.83 13.56 7.08 1151.67 25.43 56.09 1.41 0.17 0.04 0.00 15.40 4.07 0.19 51.57
nvme1n1 0.00 45.07 0.00 0.41 0.00 3.93 0.00 8.03 0.00 0.33 0.01 0.00 9.30 0.14 0.61
dm-0 0.00 48.93 0.00 0.41 0.00 0.00 0.00 0.00 0.00 0.83 0.04 0.00 8.58 0.13 0.61
sdc 170.43 4.40 0.91 0.09 0.00 0.00 0.00 0.00 5.46 1.75 0.79 5.47 21.61 1.97 34.47
sdg 198.73 10.17 7.46 0.77 0.00 0.00 0.00 0.00 7.18 9.23 1.35 38.43 77.85 2.28 47.69
sdf 26.00 3.53 0.35 0.10 0.00 0.00 0.00 0.00 10.42 1.17 0.23 13.86 29.17 3.90 11.51
sdb 95.97 9.43 0.70 0.23 0.00 0.00 0.00 0.00 14.34 1.21 1.20 7.42 25.26 4.97 52.39
sdi 161.33 204.10 0.83 7.24 0.00 0.50 0.00 0.24 33.24 103.88 26.02 5.28 36.32 1.84 67.13
sdj 80.10 8.97 0.67 0.26 0.00 0.00 0.00 0.00 15.55 3.42 1.12 8.54 30.04 4.48 39.89
sdh 26.73 6.07 0.41 0.14 0.00 0.00 0.00 0.00 10.11 0.82 0.22 15.76 24.24 3.96 12.99
sde 21.33 9.03 0.44 0.27 0.00 0.00 0.00 0.00 17.83 9.31 0.42 20.99 30.61 4.54 13.77
sdk 124.57 12.30 2.58 0.27 0.00 0.00 0.00 0.00 19.64 6.85 2.36 21.22 22.29 4.52 61.85
sdl 223.90 77.10 1.13 9.98 0.00 0.03 0.00 0.04 13.83 30.04 5.16 5.16 132.60 2.20 66.25
sda 272.10 10.93 1.22 0.29 0.00 0.00 0.00 0.00 7.50 2.56 1.65 4.60 26.99 1.30 36.71
sdd 156.20 119.07 1.55 7.04 0.00 1.00 0.00 0.83 18.62 21.37 5.11 10.13 60.59 2.31 63.57
cas1-1 445.80 16.60 1.93 0.10 0.00 0.00 0.00 0.00 11.48 1.65 5.15 4.44 6.33 0.91 41.89
cas1-2 170.07 22.23 1.04 0.13 0.00 0.00 0.00 0.00 22.53 106.88 6.21 6.29 6.09 3.79 72.85
cas1-3 231.30 11.53 1.20 0.05 0.00 0.00 0.00 0.00 5.97 102.18 2.56 5.29 4.81 1.93 46.93
cas1-4 222.33 170.30 1.83 6.13 0.00 0.00 0.00 0.00 31.96 389.60 73.45 8.43 36.87 1.91 75.09
cas1-5 75.47 16.90 0.70 0.13 0.00 0.00 0.00 0.00 8.47 2.42 0.68 9.51 8.07 1.84 17.01
cas1-6 68.90 14.37 0.54 0.10 0.00 0.00 0.00 0.00 55.15 13.50 3.99 8.07 7.21 3.00 25.01
cas1-7 334.43 32.00 8.55 0.72 0.00 0.00 0.00 0.00 7.73 85.94 5.34 26.19 23.07 1.75 64.25
cas1-8 96.03 12.07 0.75 0.09 0.00 0.00 0.00 0.00 6.72 11.58 0.78 8.02 8.00 2.06 22.27
cas1-9 228.30 611.20 1.18 7.51 0.00 0.00 0.00 0.00 26.12 789.22 488.33 5.27 12.58 0.80 67.19
cas1-10 349.80 23.97 1.81 0.17 0.00 0.00 0.00 0.00 18.00 2.44 6.35 5.29 7.13 1.45 54.28
cas1-11 427.60 25.27 3.81 0.15 0.00 0.00 0.00 0.00 10.69 23.83 5.17 9.12 6.22 1.69 76.33
cas1-12 602.30 121.30 2.64 8.83 0.00 0.00 0.00 0.00 8.15 213.79 30.84 4.48 74.53 1.04 75.49
12/06/22 11:50:03
avg-cpu: %user %nice %system %iowait %steal %idle
36.56 0.00 10.59 26.06 0.00 26.79
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1659.07 1284.03 21.10 5.12 1854.07 23.83 52.78 1.82 0.15 0.03 0.00 13.02 4.09 0.20 58.67
nvme1n1 3.47 53.43 0.07 0.62 1.03 4.53 22.96 7.82 0.72 0.25 0.01 20.35 11.84 0.18 1.00
dm-0 4.50 57.80 0.07 0.62 0.00 0.00 0.00 0.00 0.89 0.55 0.04 15.67 10.96 0.18 1.15
sdc 87.73 10.77 0.71 0.30 0.00 0.00 0.00 0.00 8.54 2.38 0.66 8.25 28.84 2.47 24.37
sdg 123.87 16.67 7.49 0.39 0.00 0.00 0.00 0.00 13.22 7.36 1.59 61.95 24.03 3.74 52.63
sdf 38.53 9.33 0.43 0.22 0.00 0.00 0.00 0.00 12.66 1.99 0.42 11.46 24.69 3.84 18.37
sdb 138.50 9.53 5.07 0.31 0.00 0.00 0.00 0.00 16.21 2.98 2.04 37.50 32.99 4.25 62.92
sdi 55.57 198.60 0.56 10.65 0.00 0.13 0.00 0.07 70.78 59.08 15.27 10.34 54.93 2.39 60.71
sdj 68.63 7.67 0.51 0.21 0.00 0.00 0.00 0.00 10.42 1.14 0.64 7.63 28.57 3.64 27.77
sdh 37.60 11.90 0.53 0.44 0.00 0.00 0.00 0.00 15.81 2.20 0.53 14.43 37.60 3.94 19.52
sde 37.10 10.17 0.46 0.36 0.00 0.00 0.00 0.00 14.21 1.93 0.46 12.68 36.18 4.04 19.11
sdk 124.30 11.77 5.26 0.40 0.00 0.00 0.00 0.00 21.83 5.56 2.58 43.32 34.41 4.57 62.16
sdl 153.50 40.33 0.90 0.76 0.00 0.00 0.00 0.00 33.84 2.67 4.98 6.02 19.20 3.43 66.47
sda 38.70 6.40 0.62 0.17 0.00 0.00 0.00 0.00 12.34 7.33 0.45 16.53 27.83 4.12 18.57
sdd 134.20 79.30 1.30 8.89 0.00 0.43 0.00 0.54 23.72 15.08 4.07 9.91 114.77 3.22 68.71
cas1-1 129.67 29.60 1.04 0.18 0.00 0.00 0.00 0.00 8.15 47.60 2.47 8.18 6.38 1.48 23.49
cas1-2 841.40 26.07 7.84 0.20 0.00 0.00 0.00 0.00 3.55 2.64 3.05 9.55 7.85 0.74 63.91
cas1-3 157.10 28.77 1.00 0.21 0.00 0.00 0.00 0.00 6.11 0.70 0.98 6.55 7.47 1.22 22.68
cas1-4 206.23 130.70 1.62 8.78 0.00 0.00 0.00 0.00 20.52 289.28 42.04 8.02 68.78 2.21 74.59
cas1-5 139.70 31.40 0.92 0.27 0.00 0.00 0.00 0.00 3.55 1.14 0.53 6.73 8.78 1.04 17.85
cas1-6 120.13 30.63 0.84 0.17 0.00 0.00 0.00 0.00 7.16 0.54 0.88 7.12 5.55 1.62 24.39
cas1-7 291.97 41.63 8.91 0.23 0.00 0.00 0.00 0.00 8.72 2.83 2.66 31.26 5.70 1.71 57.08
cas1-8 120.67 32.40 0.87 0.33 0.00 0.00 0.00 0.00 6.63 0.62 0.82 7.34 10.43 1.07 16.39
cas1-9 305.33 280.13 1.60 9.82 0.00 0.00 0.00 0.00 15.56 1113.55 316.69 5.37 35.90 0.85 49.48
cas1-10 127.77 24.13 0.76 0.14 0.00 0.00 0.00 0.00 11.52 3.86 1.57 6.07 5.81 2.25 34.11
cas1-11 478.27 33.70 6.67 0.29 0.00 0.00 0.00 0.00 9.26 2.75 4.52 14.28 8.76 1.34 68.44
cas1-12 578.30 54.40 2.59 0.27 0.00 0.00 0.00 0.00 10.85 352.28 25.44 4.59 5.14 1.15 73.07
12/06/22 11:50:33
avg-cpu: %user %nice %system %iowait %steal %idle
35.49 0.00 10.43 29.44 0.00 24.64
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1086.80 1788.10 16.67 7.08 1240.93 22.17 53.31 1.22 0.18 0.04 0.00 15.71 4.06 0.23 64.89
nvme1n1 0.07 54.70 0.00 0.56 0.00 5.33 0.00 8.88 2.50 0.24 0.01 4.00 10.54 0.11 0.63
dm-0 0.07 59.83 0.00 0.56 0.00 0.00 0.00 0.00 2.00 0.60 0.04 4.00 9.65 0.10 0.63
sdc 1266.83 110.30 5.22 4.35 0.00 0.00 0.00 0.00 17.52 15.90 21.13 4.22 40.34 0.52 71.13
sdg 144.33 18.00 9.36 0.60 0.00 0.00 0.00 0.00 13.68 3.08 1.83 66.44 34.25 3.89 63.23
sdf 27.23 13.43 0.37 0.35 0.00 0.00 0.00 0.00 16.59 2.64 0.42 14.00 26.82 5.37 21.85
sdb 123.80 8.80 7.36 0.28 0.00 0.00 0.00 0.00 11.45 2.83 1.27 60.89 32.23 4.07 53.93
sdi 84.43 104.33 0.61 5.04 0.00 0.10 0.00 0.10 45.08 58.64 9.62 7.35 49.50 3.99 75.25
sdj 29.77 10.43 0.42 0.25 0.00 0.00 0.00 0.00 14.67 2.25 0.40 14.57 24.19 4.68 18.80
sdh 43.20 13.30 0.46 0.40 0.00 0.00 0.00 0.00 15.58 2.87 0.61 10.96 31.09 3.80 21.47
sde 90.13 15.60 0.64 0.39 0.00 0.00 0.00 0.00 12.29 5.35 1.03 7.33 25.32 4.25 44.99
sdk 156.03 15.63 6.53 0.41 0.00 0.00 0.00 0.00 18.71 4.83 2.76 42.83 27.02 3.89 66.85
sdl 362.13 26.30 1.81 0.72 0.00 0.00 0.00 0.00 40.35 3.39 13.97 5.13 27.92 2.22 86.35
sda 32.10 14.13 0.32 0.38 0.00 0.00 0.00 0.00 16.55 2.40 0.49 10.30 27.52 4.80 22.21
sdd 54.57 58.53 0.76 5.38 0.00 0.00 0.00 0.00 17.36 58.96 4.20 14.17 94.04 3.57 40.43
cas1-1 130.87 28.70 0.81 0.15 0.00 0.00 0.00 0.00 4.23 1.65 0.60 6.30 5.53 1.11 17.65
cas1-2 251.00 23.50 7.91 0.19 0.00 0.00 0.00 0.00 8.54 5.20 2.27 32.25 8.07 2.05 56.29
cas1-3 1485.60 567.97 6.19 7.93 0.00 0.00 0.00 0.00 16.96 86.85 74.52 4.27 14.30 0.36 74.19
cas1-4 188.97 70.83 1.33 5.09 0.00 0.00 0.00 0.00 5.94 360.68 26.67 7.21 73.52 1.64 42.60
cas1-5 198.30 35.53 1.15 0.19 0.00 0.00 0.00 0.00 6.31 1.83 1.32 5.92 5.59 1.88 43.85
cas1-6 139.57 28.33 0.89 0.17 0.00 0.00 0.00 0.00 4.15 0.82 0.60 6.52 6.06 0.96 16.08
cas1-7 276.90 37.03 9.94 0.35 0.00 0.00 0.00 0.00 7.92 1.13 2.24 36.77 9.59 2.05 64.20
cas1-8 151.07 29.40 1.01 0.22 0.00 0.00 0.00 0.00 8.57 0.72 1.32 6.82 7.69 0.97 17.52
cas1-9 222.70 86.27 1.34 3.89 0.00 0.00 0.00 0.00 19.94 764.42 70.38 6.17 46.22 2.44 75.25
cas1-10 118.60 30.97 0.82 0.15 0.00 0.00 0.00 0.00 9.38 0.45 1.13 7.07 5.08 1.02 15.27
cas1-11 634.70 41.90 8.47 0.24 0.00 0.00 0.00 0.00 5.70 1.29 3.67 13.66 5.89 1.08 72.88
cas1-12 497.63 31.40 2.40 0.22 0.00 0.00 0.00 0.00 33.47 4.70 16.80 4.93 7.06 1.64 86.56
12/06/22 11:51:03
avg-cpu: %user %nice %system %iowait %steal %idle
26.41 0.00 8.89 19.10 0.00 45.60
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 411.00 604.13 6.49 2.46 228.73 18.97 35.75 3.04 0.16 0.04 0.00 16.17 4.17 0.34 34.44
nvme1n1 0.17 44.03 0.00 0.37 0.00 3.70 0.00 7.75 7.80 0.26 0.01 11.20 8.60 0.13 0.56
dm-0 0.17 47.70 0.00 0.37 0.00 0.00 0.00 0.00 8.00 0.60 0.03 11.20 7.96 0.12 0.56
sdc 310.80 193.40 1.27 6.33 0.00 0.27 0.00 0.14 95.87 32.67 35.37 4.18 33.51 1.73 87.21
sdg 23.20 2.00 1.38 0.05 0.00 0.00 0.00 0.00 10.59 0.57 0.22 61.07 26.80 4.52 11.39
sdf 13.07 4.47 0.10 0.08 0.00 0.00 0.00 0.00 13.26 16.50 0.22 8.04 17.58 3.98 6.97
sdb 19.03 3.90 1.10 0.07 0.00 0.00 0.00 0.00 19.02 1.86 0.33 59.41 18.22 6.08 13.93
sdi 38.37 9.40 0.18 0.36 0.00 0.00 0.00 0.00 10.97 2.43 0.39 4.93 39.26 3.06 14.63
sdj 9.93 4.47 0.17 0.11 0.00 0.00 0.00 0.00 13.26 1.10 0.11 17.93 24.15 5.31 7.65
sdh 6.03 2.43 0.16 0.07 0.00 0.00 0.00 0.00 16.18 1.15 0.09 26.56 29.92 4.14 3.51
sde 31.53 1.17 0.17 0.02 0.00 0.00 0.00 0.00 10.78 0.49 0.30 5.54 21.60 4.06 13.28
sdk 26.57 2.37 1.51 0.05 0.00 0.00 0.00 0.00 12.65 2.13 0.31 58.18 22.08 3.94 11.40
sdl 34.60 3.13 0.20 0.07 0.00 0.00 0.00 0.00 28.08 1.07 0.90 5.87 22.77 3.20 12.08
sda 14.67 1.17 0.32 0.03 0.00 0.00 0.00 0.00 11.96 1.49 0.15 22.55 25.83 3.19 5.05
sdd 10.97 3.47 0.17 0.26 0.00 0.30 0.00 7.96 9.69 2.75 0.09 15.81 77.54 3.08 4.44
cas1-1 24.50 3.33 0.36 0.02 0.00 0.00 0.00 0.00 466.49 0.32 11.43 15.22 6.88 18.86 52.48
cas1-2 39.37 7.77 1.19 0.03 0.00 0.00 0.00 0.00 98.76 3965.06 34.68 30.98 4.41 3.70 17.44
cas1-3 349.57 254.07 1.44 2.81 0.00 0.00 0.00 0.00 86.94 4938.89 1285.20 4.23 11.34 1.58 95.24
cas1-4 33.80 9.07 0.27 0.27 0.00 0.00 0.00 0.00 27.29 1.94 0.94 8.17 30.29 2.07 8.87
cas1-5 62.70 4.47 0.30 0.02 0.00 0.00 0.00 0.00 20.44 0.09 1.28 4.96 4.63 4.01 26.95
cas1-6 35.50 15.20 0.20 0.07 0.00 0.00 0.00 0.00 5.84 8.71 0.34 5.75 4.86 1.69 8.55
cas1-7 49.83 4.00 1.48 0.02 0.00 0.00 0.00 0.00 82.27 0.17 4.10 30.45 4.90 13.73 73.93
cas1-8 15.67 9.43 0.20 0.09 0.00 0.00 0.00 0.00 113.54 1.71 1.79 13.06 9.31 1.45 3.64
cas1-9 76.67 12.43 0.37 0.23 0.00 0.00 0.00 0.00 322.38 7.17 24.81 4.96 19.10 4.51 40.15
cas1-10 28.67 4.70 0.25 0.02 0.00 0.00 0.00 0.00 117.52 212.26 4.37 9.05 5.28 2.57 8.59
cas1-11 90.43 8.80 1.75 0.05 0.00 0.00 0.00 0.00 23.65 3.33 2.17 19.86 5.89 1.43 14.23
cas1-12 58.93 7.40 0.31 0.03 0.00 0.00 0.00 0.00 71.50 4.09 4.24 5.47 4.76 1.68 11.13
12/06/22 11:51:33
avg-cpu: %user %nice %system %iowait %steal %idle
32.58 0.00 9.84 27.55 0.00 30.02
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 925.63 967.53 13.82 3.89 1330.97 23.87 58.98 2.41 0.17 0.03 0.00 15.29 4.12 0.29 54.08
nvme1n1 0.00 44.63 0.00 0.56 0.00 3.53 0.00 7.34 0.00 0.25 0.01 0.00 12.83 0.14 0.61
dm-0 0.00 47.87 0.00 0.56 0.00 0.00 0.00 0.00 0.00 0.73 0.03 0.00 11.98 0.13 0.61
sdc 89.83 205.93 0.56 11.52 0.00 0.70 0.00 0.34 83.33 113.20 30.33 6.35 57.30 2.25 66.61
sdg 98.30 7.47 4.92 0.19 0.00 0.00 0.00 0.00 12.49 1.76 1.10 51.29 25.61 4.00 42.28
sdf 93.87 14.53 0.63 0.36 0.00 0.00 0.00 0.00 5.36 6.62 0.50 6.86 25.29 2.77 30.03
sdb 133.73 5.90 6.04 0.19 0.00 0.00 0.00 0.00 11.97 3.12 1.44 46.25 33.83 3.69 51.51
sdi 73.67 13.50 0.53 0.36 0.00 0.00 0.00 0.00 19.17 3.09 1.34 7.31 27.64 3.01 26.20
sdj 84.17 7.00 0.62 0.18 0.00 0.00 0.00 0.00 18.74 2.29 1.43 7.57 26.53 3.53 32.21
sdh 46.40 5.23 0.54 0.14 0.00 0.00 0.00 0.00 13.70 1.43 0.55 11.85 27.31 3.44 17.79
sde 47.50 4.60 0.37 0.21 0.00 0.00 0.00 0.00 11.97 1.23 0.48 7.88 46.96 4.19 21.84
sdk 136.10 10.77 5.81 0.29 0.00 0.00 0.00 0.00 20.80 2.15 2.64 43.72 27.62 4.06 59.60
sdl 262.27 6.17 1.24 0.23 0.00 0.00 0.00 0.00 19.67 6.64 4.73 4.83 37.79 2.19 58.71
sda 137.93 6.70 0.68 0.25 0.00 0.03 0.00 0.50 3.77 1.60 0.43 5.06 37.95 1.57 22.72
sdd 73.90 10.00 1.23 0.31 0.00 0.00 0.00 0.00 17.56 1.53 1.19 17.00 31.75 4.25 35.69
cas1-1 262.20 15.73 1.20 0.18 0.00 0.00 0.00 0.00 3.98 180.88 3.89 4.69 11.44 1.16 32.24
cas1-2 244.47 15.33 6.50 0.14 0.00 0.00 0.00 0.00 16.34 11.86 4.18 27.23 9.04 2.22 57.56
cas1-3 196.70 227.97 1.40 12.35 0.00 0.00 0.00 0.00 38.13 2044.02 473.47 7.31 55.49 1.40 59.53
cas1-4 224.67 24.93 1.84 0.22 0.00 0.00 0.00 0.00 12.29 7.54 2.95 8.37 8.86 1.78 44.39
cas1-5 160.87 15.03 0.87 0.19 0.00 0.00 0.00 0.00 7.53 308.00 5.84 5.54 13.14 2.09 36.71
cas1-6 191.70 30.57 1.06 0.17 0.00 0.00 0.00 0.00 3.91 2.30 0.82 5.64 5.79 1.29 28.75
cas1-7 223.57 19.80 5.48 0.10 0.00 0.00 0.00 0.00 8.57 7.86 2.07 25.12 5.36 1.96 47.67
cas1-8 159.23 12.13 1.00 0.09 0.00 0.00 0.00 0.00 9.66 1.66 1.56 6.42 7.53 1.54 26.41
cas1-9 167.23 27.73 0.98 0.19 0.00 0.00 0.00 0.00 12.58 15.33 2.53 6.02 7.00 1.73 33.79
cas1-10 362.00 22.33 1.75 0.12 0.00 0.00 0.00 0.00 5.44 0.62 1.98 4.95 5.50 0.81 31.29
cas1-11 614.03 24.70 7.71 0.13 0.00 0.00 0.00 0.00 9.55 1.83 5.91 12.86 5.38 0.98 62.60
cas1-12 371.03 17.70 1.70 0.15 0.00 0.00 0.00 0.00 20.79 209.48 11.42 4.69 8.93 1.79 69.73
12/06/22 11:52:03
avg-cpu: %user %nice %system %iowait %steal %idle
35.00 0.00 10.38 31.35 0.00 23.27
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1156.63 1040.10 18.20 4.13 2311.80 15.33 66.65 1.45 0.20 0.04 0.01 16.11 4.07 0.28 61.39
nvme1n1 1.03 44.50 0.02 0.41 0.20 4.67 16.22 9.49 2.42 0.27 0.01 17.03 9.51 0.16 0.71
dm-0 1.23 49.03 0.02 0.41 0.00 0.00 0.00 0.00 2.27 0.66 0.04 14.27 8.65 0.14 0.71
sdc 152.90 58.63 5.53 5.68 0.00 0.00 0.00 0.00 20.75 43.42 5.46 37.05 99.27 2.65 56.11
sdg 146.50 11.27 5.95 0.32 0.00 0.00 0.00 0.00 16.24 3.69 2.21 41.57 29.21 4.21 66.48
sdf 207.03 7.97 1.06 0.22 0.00 0.00 0.00 0.00 6.36 1.42 1.11 5.24 28.18 1.77 38.07
sdb 42.63 4.90 0.57 0.20 0.00 0.00 0.00 0.00 12.72 0.87 0.47 13.65 41.55 4.12 19.59
sdi 121.83 13.70 0.78 0.39 0.00 0.00 0.00 0.00 8.51 4.34 0.94 6.54 29.47 2.78 37.64
sdj 40.07 7.40 0.44 0.20 0.00 0.00 0.00 0.00 13.05 1.64 0.45 11.30 27.28 4.02 19.08
sdh 37.27 10.30 0.52 0.31 0.00 0.00 0.00 0.00 13.22 1.52 0.43 14.41 30.42 4.40 20.92
sde 375.33 71.00 1.71 1.14 0.00 0.00 0.00 0.00 48.44 53.71 21.26 4.68 16.47 1.83 81.71
sdk 144.30 6.77 2.66 0.21 0.00 0.00 0.00 0.00 22.51 2.03 3.02 18.86 32.49 5.56 83.92
sdl 112.47 9.53 0.69 0.19 0.00 0.00 0.00 0.00 5.99 1.18 0.57 6.24 20.73 1.87 22.77
sda 58.40 10.07 0.57 0.33 0.00 0.00 0.00 0.00 15.72 2.23 0.82 9.96 33.51 3.64 24.93
sdd 363.90 13.63 1.78 0.36 0.00 0.00 0.00 0.00 11.97 19.22 3.91 5.00 27.10 1.68 63.31
cas1-1 205.23 26.93 1.19 0.22 0.00 0.00 0.00 0.00 7.24 7.87 1.70 5.92 8.42 1.20 27.89
cas1-2 131.90 8.60 0.96 0.11 0.00 0.00 0.00 0.00 5.11 3.46 0.70 7.47 12.95 1.31 18.39
cas1-3 305.33 131.13 6.17 5.27 0.00 0.00 0.00 0.00 11.66 482.14 66.78 20.69 41.19 1.29 56.20
cas1-4 505.73 33.30 2.53 0.18 0.00 0.00 0.00 0.00 9.54 1.13 4.86 5.13 5.64 1.19 63.88
cas1-5 809.73 161.37 3.44 0.92 0.00 0.00 0.00 0.00 24.90 252.72 60.95 4.35 5.85 0.85 82.13
cas1-6 1012.77 24.87 4.26 0.15 0.00 0.00 0.00 0.00 1.70 1.47 1.75 4.31 6.04 0.38 39.33
cas1-7 583.00 30.43 7.67 0.19 0.00 0.00 0.00 0.00 5.29 2.20 3.15 13.48 6.41 1.15 70.76
cas1-8 128.93 23.77 0.90 0.17 0.00 0.00 0.00 0.00 5.73 2.10 0.79 7.17 7.41 1.31 20.00
cas1-9 215.57 40.77 1.16 0.27 0.00 0.00 0.00 0.00 5.36 3.17 1.29 5.53 6.70 1.61 41.29
cas1-10 143.83 17.03 0.92 0.09 0.00 0.00 0.00 0.00 4.14 0.72 0.61 6.56 5.56 1.16 18.59
cas1-11 705.50 19.00 4.92 0.16 0.00 0.00 0.00 0.00 6.03 3.68 4.33 7.14 8.72 1.16 83.93
cas1-12 190.43 28.27 1.04 0.14 0.00 0.00 0.00 0.00 5.87 1.39 1.16 5.62 5.12 0.96 20.92
12/06/22 11:52:33
avg-cpu: %user %nice %system %iowait %steal %idle
30.24 0.00 9.49 28.54 0.00 31.73
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 784.30 831.90 12.51 3.35 1344.90 20.70 63.16 2.43 0.19 0.04 0.01 16.34 4.12 0.31 50.09
nvme1n1 0.23 48.70 0.00 0.53 0.00 3.63 0.00 6.94 2.29 0.21 0.00 21.71 11.16 0.14 0.67
dm-0 0.23 52.20 0.00 0.53 0.00 0.00 0.00 0.00 2.29 0.73 0.04 21.71 10.43 0.13 0.67
sdc 66.50 65.90 4.02 5.41 0.00 0.77 0.00 1.15 37.68 16.17 3.45 61.97 84.02 3.02 40.01
sdg 153.97 9.53 3.52 0.26 0.00 0.00 0.00 0.00 23.82 2.34 3.48 23.42 27.83 3.56 58.24
sdf 26.70 8.73 0.30 0.20 0.00 0.00 0.00 0.00 12.16 0.94 0.28 11.65 23.92 3.84 13.61
sdb 63.93 9.83 0.37 1.01 0.00 0.00 0.00 0.00 13.77 6.26 0.80 5.97 105.04 3.80 28.05
sdi 42.70 12.43 0.35 0.34 0.00 0.00 0.00 0.00 31.36 2.27 1.28 8.46 28.15 4.01 22.09
sdj 73.30 6.83 0.40 0.25 0.00 0.00 0.00 0.00 10.61 1.47 0.71 5.57 37.97 2.65 21.21
sdh 74.93 5.77 0.48 0.17 0.00 0.00 0.00 0.00 6.25 1.40 0.40 6.55 30.10 3.09 24.97
sde 282.37 98.17 1.41 5.44 0.00 0.13 0.00 0.14 44.94 66.07 18.55 5.10 56.74 1.57 59.85
sdk 78.90 7.33 0.62 0.26 0.00 0.00 0.00 0.00 24.65 3.32 1.83 8.07 36.07 6.99 60.28
sdl 22.53 7.13 0.37 0.23 0.00 0.00 0.00 0.00 12.45 1.50 0.25 16.93 32.90 5.24 15.55
sda 87.47 2.37 0.53 0.06 0.00 0.00 0.00 0.00 17.52 0.92 1.39 6.16 25.24 2.50 22.48
sdd 248.97 7.27 1.33 0.18 0.00 0.00 0.00 0.00 9.65 1.39 1.98 5.45 25.87 1.63 41.67
cas1-1 338.80 11.07 1.57 0.05 0.00 0.00 0.00 0.00 5.92 2.42 2.03 4.74 5.00 0.89 31.16
cas1-2 433.87 18.67 1.85 0.92 0.00 0.00 0.00 0.00 5.60 11.50 2.65 4.36 50.36 0.61 27.40
cas1-3 118.83 22.93 4.25 1.34 0.00 0.00 0.00 0.00 24.13 3929.23 92.98 36.59 59.99 3.29 46.69
cas1-4 309.30 13.77 1.57 0.07 0.00 0.00 0.00 0.00 35.00 16.64 11.05 5.21 5.51 1.76 56.87
cas1-5 374.30 270.40 1.78 8.09 0.00 0.00 0.00 0.00 58.17 361.75 119.59 4.87 30.63 1.03 66.39
cas1-6 70.87 19.60 0.49 0.10 0.00 0.00 0.00 0.00 13.08 3.26 0.99 7.06 5.12 2.06 18.61
cas1-7 564.73 19.37 5.16 0.12 0.00 0.00 0.00 0.00 9.86 14.44 5.85 9.36 6.15 1.15 67.35
cas1-8 139.07 11.97 0.75 0.09 0.00 0.00 0.00 0.00 8.82 0.78 1.24 5.51 8.09 2.27 34.31
cas1-9 121.17 22.70 0.69 0.15 0.00 0.00 0.00 0.00 16.59 15.91 2.37 5.79 6.71 2.04 29.41
cas1-10 165.07 16.07 0.84 0.16 0.00 0.00 0.00 0.00 4.67 4.48 0.84 5.22 10.28 1.24 22.47
cas1-11 312.37 17.20 1.57 0.15 0.00 0.00 0.00 0.00 10.68 18.43 3.65 5.15 8.89 2.38 78.49
cas1-12 72.93 14.87 0.58 0.10 0.00 0.00 0.00 0.00 10.07 17.33 0.99 8.10 6.82 2.21 19.37
12/06/22 11:53:03
avg-cpu: %user %nice %system %iowait %steal %idle
33.83 0.00 10.22 28.31 0.00 27.64
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1090.70 1278.33 18.18 5.12 2338.80 27.93 68.20 2.14 0.21 0.03 0.00 17.07 4.10 0.25 59.19
nvme1n1 0.33 45.80 0.01 0.46 0.00 3.63 0.00 7.35 2.40 0.18 0.00 16.00 10.22 0.15 0.68
dm-0 0.33 49.40 0.01 0.46 0.00 0.00 0.00 0.00 2.40 0.71 0.04 16.00 9.49 0.14 0.68
sdc 61.93 27.20 0.63 0.69 0.00 0.00 0.00 0.00 22.43 8.22 1.46 10.42 26.08 3.27 29.11
sdg 195.07 9.87 4.24 0.23 0.00 0.00 0.00 0.00 24.42 2.68 4.53 22.25 24.34 3.69 75.64
sdf 57.30 11.17 0.49 0.29 0.00 0.03 0.00 0.30 14.19 2.04 0.72 8.84 26.28 4.79 32.79
sdb 120.60 12.50 0.88 1.63 0.00 0.23 0.00 1.83 16.99 32.54 2.21 7.47 133.44 4.88 65.00
sdi 104.37 13.97 0.72 0.35 0.00 0.00 0.00 0.00 12.82 4.29 1.23 7.04 25.44 4.15 49.11
sdj 72.30 10.73 0.63 0.31 0.00 0.00 0.00 0.00 16.15 3.98 1.07 8.91 29.88 2.92 24.28
sdh 53.10 10.37 0.49 0.43 0.00 0.20 0.00 1.89 9.08 2.71 0.42 9.42 42.69 3.14 19.91
sde 66.73 167.73 0.60 8.65 0.03 2.40 0.05 1.41 73.76 107.96 22.61 9.28 52.82 2.39 56.03
sdk 134.27 6.53 0.88 0.27 0.00 0.00 0.00 0.00 12.79 2.30 1.54 6.71 42.90 4.66 65.67
sdl 217.37 61.27 1.79 1.13 0.00 0.00 0.00 0.00 9.65 16.30 2.80 8.41 18.87 1.30 36.11
sda 85.60 14.03 0.70 0.38 0.00 0.00 0.00 0.00 11.92 1.87 0.88 8.39 27.49 3.11 31.03
sdd 44.93 6.10 0.42 0.23 0.00 0.00 0.00 0.00 10.50 3.86 0.41 9.53 38.40 3.29 16.80
cas1-1 404.77 36.60 2.03 0.21 0.00 0.00 0.00 0.00 4.15 3.17 1.80 5.13 6.00 0.79 35.00
cas1-2 702.47 34.67 3.21 1.54 0.00 0.00 0.00 0.00 8.24 14.11 6.28 4.69 45.37 0.94 69.05
cas1-3 187.13 53.33 1.14 0.35 0.00 0.00 0.00 0.00 8.62 9.99 2.15 6.25 6.77 1.35 32.56
cas1-4 177.63 21.57 0.97 0.17 0.00 0.00 0.00 0.00 3.53 14.89 0.95 5.59 8.20 1.40 27.84
cas1-5 181.13 254.33 1.13 5.74 0.00 0.00 0.00 0.00 27.67 1112.69 288.01 6.39 23.13 1.21 52.80
cas1-6 200.83 25.03 1.07 0.17 0.00 0.00 0.00 0.00 5.52 8.79 1.33 5.47 7.01 1.80 40.61
cas1-7 669.23 26.07 6.26 0.16 0.00 0.00 0.00 0.00 12.10 20.36 8.63 9.58 6.30 1.19 82.43
cas1-8 158.83 33.97 0.94 0.39 0.00 0.00 0.00 0.00 7.79 2.76 1.33 6.07 11.72 1.41 27.11
cas1-9 223.20 33.10 1.25 0.18 0.00 0.00 0.00 0.00 27.84 19.29 6.85 5.75 5.66 2.13 54.64
cas1-10 584.93 24.17 2.67 0.18 0.00 0.00 0.00 0.00 5.65 2.01 3.35 4.67 7.74 0.46 27.84
cas1-11 464.50 19.97 2.25 0.21 0.00 0.00 0.00 0.00 5.71 7.15 2.80 4.95 10.67 1.58 76.61
cas1-12 346.87 160.00 2.35 1.11 0.00 0.00 0.00 0.00 7.83 384.01 64.16 6.95 7.11 0.80 40.32
12/06/22 11:53:33
avg-cpu: %user %nice %system %iowait %steal %idle
35.79 0.00 10.60 29.46 0.00 24.15
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1114.43 1143.07 16.96 4.56 1599.63 18.63 58.94 1.60 0.18 0.03 0.00 15.59 4.09 0.26 58.88
nvme1n1 0.17 49.27 0.00 0.48 0.00 3.43 0.00 6.51 0.20 0.22 0.00 8.00 9.89 0.14 0.71
dm-0 0.17 52.63 0.00 0.48 0.00 0.00 0.00 0.00 0.00 0.65 0.03 8.00 9.27 0.13 0.71
sdc 38.30 8.50 0.55 0.23 0.00 0.00 0.00 0.00 14.05 9.03 0.54 14.59 28.24 4.11 19.21
sdg 200.30 19.03 6.54 0.50 0.00 0.00 0.00 0.00 25.57 8.56 4.99 33.45 27.05 4.11 90.25
sdf 81.03 7.13 0.91 0.27 0.00 0.00 0.00 0.00 7.67 1.14 0.53 11.48 39.07 3.86 34.00
sdb 216.17 30.60 1.21 0.65 0.00 0.00 0.00 0.00 16.11 4.22 3.33 5.73 21.69 2.64 65.15
sdi 38.27 10.00 0.58 0.30 0.00 0.00 0.00 0.00 14.76 2.21 0.51 15.60 31.08 4.33 20.91
sdj 817.20 13.30 3.44 0.42 0.00 0.00 0.00 0.00 13.35 4.69 9.34 4.31 32.06 0.99 81.89
sdh 50.67 14.23 0.53 0.39 0.00 0.00 0.00 0.00 16.47 1.67 0.74 10.79 27.87 4.14 26.88
sde 100.97 51.00 0.79 7.12 0.00 0.00 0.00 0.00 20.98 45.19 4.16 8.01 142.90 4.99 75.89
sdk 198.10 17.20 1.20 0.51 0.00 0.07 0.00 0.39 8.46 2.52 1.54 6.20 30.23 2.70 58.05
sdl 184.43 75.53 1.16 4.42 0.00 0.17 0.00 0.22 19.75 36.73 6.03 6.44 59.94 2.85 74.19
sda 51.80 13.40 0.45 0.39 0.00 0.00 0.00 0.00 15.01 1.30 0.68 8.95 29.44 3.84 25.07
sdd 57.40 16.60 0.92 0.43 0.00 0.00 0.00 0.00 24.01 4.62 1.34 16.36 26.24 3.77 27.87
cas1-1 150.93 23.67 0.90 0.16 0.00 0.00 0.00 0.00 9.35 52.10 2.64 6.08 6.74 1.47 25.68
cas1-2 446.53 69.20 2.23 0.34 0.00 0.00 0.00 0.00 8.37 3.42 3.97 5.12 5.02 1.31 67.80
cas1-3 137.17 22.97 1.07 0.12 0.00 0.00 0.00 0.00 4.17 3.76 0.66 7.97 5.57 1.20 19.17
cas1-4 209.53 33.43 1.55 0.21 0.00 0.00 0.00 0.00 10.48 3.72 2.32 7.57 6.58 1.20 29.24
cas1-5 235.77 61.53 1.47 6.86 0.00 0.00 0.00 0.00 23.35 748.33 51.55 6.38 114.16 2.61 77.63
cas1-6 211.10 21.93 1.74 0.20 0.00 0.00 0.00 0.00 3.48 5.28 0.85 8.44 9.54 1.60 37.37
cas1-7 714.23 45.90 8.68 0.24 0.00 0.00 0.00 0.00 7.78 38.14 7.31 12.45 5.38 1.20 91.13
cas1-8 173.53 30.93 1.24 0.17 0.00 0.00 0.00 0.00 4.68 1.92 0.87 7.29 5.62 1.07 21.95
cas1-9 155.20 27.37 1.11 0.18 0.00 0.00 0.00 0.00 4.50 0.60 0.71 7.30 6.70 1.29 23.53
cas1-10 1227.93 31.70 5.13 0.28 0.00 0.00 0.00 0.00 10.91 1.60 13.45 4.28 9.14 0.69 87.45
cas1-11 271.47 37.20 1.55 0.28 0.00 0.00 0.00 0.00 7.14 7.11 2.20 5.85 7.76 1.99 61.40
cas1-12 323.60 156.50 1.96 3.92 0.00 0.00 0.00 0.00 14.12 150.27 28.09 6.20 25.66 1.47 70.48
12/06/22 11:54:03
avg-cpu: %user %nice %system %iowait %steal %idle
34.54 0.00 10.58 29.08 0.00 25.81
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 921.13 1518.23 15.12 6.01 1610.30 18.23 63.61 1.19 0.20 0.03 0.00 16.81 4.06 0.25 60.91
nvme1n1 0.17 63.00 0.00 0.58 0.00 3.93 0.00 5.88 4.80 0.20 0.01 15.20 9.48 0.10 0.64
dm-0 0.17 66.80 0.00 0.58 0.00 0.00 0.00 0.00 4.80 0.43 0.03 15.20 8.95 0.10 0.65
sdc 152.93 17.83 0.89 0.95 0.00 0.10 0.00 0.56 5.91 18.44 1.10 5.98 54.80 1.74 29.68
sdg 538.37 45.80 8.74 1.47 0.00 0.00 0.00 0.00 13.65 33.36 7.81 16.63 32.88 1.26 73.56
sdf 146.30 13.83 0.85 0.33 0.00 0.00 0.00 0.00 11.05 2.81 1.47 5.97 24.10 3.48 55.68
sdb 120.57 18.73 0.71 0.39 0.00 0.00 0.00 0.00 29.70 3.86 3.39 6.03 21.25 5.29 73.69
sdi 44.67 10.30 0.45 0.28 0.00 0.00 0.00 0.00 16.52 3.23 0.67 10.39 27.79 3.57 19.60
sdj 180.00 41.97 1.05 1.21 0.00 0.03 0.00 0.08 21.50 12.06 4.08 5.97 29.57 3.40 75.48
sdh 33.33 14.87 0.43 0.40 0.00 0.00 0.00 0.00 14.65 3.56 0.45 13.21 27.39 4.28 20.61
sde 32.07 56.53 0.56 3.46 0.00 0.43 0.00 0.76 48.72 85.56 6.27 17.98 62.62 3.63 32.12
sdk 122.83 9.13 0.73 0.34 0.00 0.00 0.00 0.00 13.96 1.55 1.55 6.09 38.19 5.31 70.08
sdl 185.47 110.60 1.32 8.68 0.00 1.67 0.00 1.48 12.44 46.10 7.08 7.27 80.35 1.76 51.99
sda 38.87 9.37 0.95 0.27 0.00 0.00 0.00 0.00 13.70 2.02 0.47 25.10 29.04 4.19 20.23
sdd 51.63 12.27 0.47 0.32 0.00 0.00 0.00 0.00 13.46 1.60 0.61 9.33 26.74 2.93 18.71
cas1-1 166.83 25.27 1.48 0.15 0.00 0.00 0.00 0.00 5.34 1.89 0.94 9.09 6.12 1.03 19.87
cas1-2 304.37 41.67 1.53 0.21 0.00 0.00 0.00 0.00 13.69 5.90 4.41 5.15 5.17 2.28 78.91
cas1-3 275.10 52.27 1.46 0.81 0.00 0.00 0.00 0.00 3.49 19.12 1.96 5.42 15.84 0.88 28.87
cas1-4 163.27 34.57 1.19 0.19 0.00 0.00 0.00 0.00 4.08 4.47 0.82 7.49 5.51 0.97 19.21
cas1-5 156.10 81.93 1.07 3.30 0.00 0.00 0.00 0.00 14.90 830.69 70.39 7.03 41.26 1.20 28.67
cas1-6 341.63 34.00 1.66 0.20 0.00 0.00 0.00 0.00 5.88 2.05 2.08 4.98 6.03 1.63 61.16
cas1-7 739.23 289.33 9.57 1.66 0.00 0.00 0.00 0.00 12.54 19.42 14.89 13.26 5.87 0.78 79.97
cas1-8 151.47 34.23 0.92 0.23 0.00 0.00 0.00 0.00 4.01 0.72 0.63 6.22 6.75 1.33 24.65
cas1-9 155.37 30.63 0.94 0.19 0.00 0.00 0.00 0.00 5.87 1.21 0.95 6.21 6.37 1.06 19.80
cas1-10 739.73 103.50 3.32 1.05 0.00 0.00 0.00 0.00 6.73 100.13 15.34 4.60 10.38 0.93 78.83
cas1-11 326.67 23.93 1.54 0.24 0.00 0.00 0.00 0.00 6.74 4.12 2.30 4.84 10.20 2.19 76.80
cas1-12 307.93 170.10 1.82 8.46 0.00 0.00 0.00 0.00 8.50 521.18 91.27 6.05 50.91 1.10 52.43
12/06/22 11:54:33
avg-cpu: %user %nice %system %iowait %steal %idle
32.41 0.00 9.95 26.27 0.00 31.37
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 804.10 1120.23 13.56 4.52 1529.87 32.83 65.55 2.85 0.20 0.04 0.00 17.26 4.13 0.27 52.40
nvme1n1 0.00 59.80 0.00 0.48 0.00 3.77 0.00 5.93 0.00 0.26 0.01 0.00 8.28 0.10 0.60
dm-0 0.00 63.53 0.00 0.48 0.00 0.00 0.00 0.00 0.00 0.63 0.04 0.00 7.80 0.09 0.60
sdc 55.90 13.70 0.43 0.33 0.00 0.00 0.00 0.00 18.01 3.01 0.92 7.79 24.81 3.50 24.36
sdg 96.57 212.87 6.60 9.11 0.00 1.03 0.00 0.48 56.85 240.28 56.10 69.96 43.84 2.29 70.97
sdf 44.37 13.90 0.36 0.40 0.00 0.10 0.00 0.71 13.93 2.47 0.55 8.42 29.22 3.65 21.29
sdb 58.13 8.63 0.51 0.24 0.00 0.00 0.00 0.00 13.07 1.32 0.66 9.03 27.98 3.66 24.41
sdi 52.53 11.43 0.43 0.30 0.00 0.00 0.00 0.00 13.31 2.01 0.60 8.38 26.94 3.27 20.92
sdj 71.27 58.87 0.54 5.54 0.00 0.57 0.00 0.95 13.82 23.88 2.17 7.76 96.44 2.70 35.15
sdh 266.87 8.63 1.26 0.24 0.00 0.00 0.00 0.00 11.78 25.77 2.91 4.82 28.53 1.33 36.73
sde 47.80 22.23 0.42 0.45 0.00 0.00 0.00 0.00 16.42 2.47 0.71 8.92 20.73 3.65 25.53
sdk 121.00 4.77 0.87 0.15 0.00 0.00 0.00 0.00 13.19 4.39 1.43 7.35 31.44 5.21 65.48
sdl 42.17 83.03 0.50 11.09 0.00 0.67 0.00 0.80 33.72 29.27 3.63 12.11 136.73 2.61 32.68
sda 373.03 12.90 1.77 0.35 0.00 0.00 0.00 0.00 12.79 4.96 4.07 4.87 28.17 1.30 50.28
sdd 36.40 7.77 0.72 0.20 0.00 0.00 0.00 0.00 12.96 1.07 0.41 20.37 26.32 4.02 17.77
cas1-1 478.33 37.40 2.28 0.26 0.00 0.00 0.00 0.00 17.03 10.14 8.52 4.87 7.09 1.05 53.97
cas1-2 199.10 27.60 1.11 0.17 0.00 0.00 0.00 0.00 5.81 25.21 1.85 5.69 6.19 1.70 38.44
cas1-3 167.27 31.20 0.87 0.19 0.00 0.00 0.00 0.00 13.80 19.87 2.93 5.34 6.14 1.98 39.36
cas1-4 131.60 17.23 1.15 0.09 0.00 0.00 0.00 0.00 5.59 3.36 0.79 8.98 5.34 1.04 15.44
cas1-5 160.93 48.53 0.93 0.27 0.00 0.00 0.00 0.00 12.53 4.57 2.24 5.93 5.66 1.31 27.36
cas1-6 177.40 40.97 0.91 0.29 0.00 0.00 0.00 0.00 9.29 104.53 5.93 5.27 7.32 1.25 27.39
cas1-7 215.63 251.07 7.09 8.08 0.00 0.00 0.00 0.00 27.50 1534.18 391.11 33.68 32.95 1.52 70.73
cas1-8 492.07 27.07 2.18 0.21 0.00 0.00 0.00 0.00 7.13 240.20 10.01 4.53 7.98 0.76 39.59
cas1-9 168.13 25.13 0.93 0.17 0.00 0.00 0.00 0.00 8.61 1.27 1.48 5.66 7.09 1.08 20.92
cas1-10 284.17 116.50 1.40 6.46 0.00 0.00 0.00 0.00 8.02 164.99 21.50 5.03 56.75 1.06 42.61
cas1-11 625.03 13.50 2.87 0.11 0.00 0.00 0.00 0.00 4.25 50.15 3.33 4.69 8.03 1.23 78.49
cas1-12 166.17 102.37 1.00 10.94 0.00 0.00 0.00 0.00 13.78 726.05 76.61 6.16 109.39 1.25 33.69
12/06/22 11:55:03
avg-cpu: %user %nice %system %iowait %steal %idle
34.34 0.00 10.77 20.14 0.00 34.75
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 1091.33 1050.43 15.36 4.19 1598.13 18.53 59.42 1.73 0.18 0.04 0.00 14.41 4.08 0.27 57.59
nvme1n1 1.97 75.00 0.05 0.65 4.13 4.87 67.76 6.09 0.64 0.36 0.01 26.17 8.93 0.13 1.01
dm-0 6.10 79.73 0.05 0.65 0.00 0.00 0.00 0.00 0.28 0.68 0.06 8.44 8.41 0.12 1.01
sdc 49.40 13.97 0.46 0.38 0.00 0.00 0.00 0.00 13.69 8.79 0.70 9.55 28.06 4.01 25.41
sdg 108.30 114.97 9.80 13.54 0.00 0.30 0.00 0.26 25.07 146.77 19.27 92.68 120.59 2.74 61.16
sdf 34.53 14.90 0.31 0.40 0.00 0.00 0.00 0.00 15.95 2.30 0.51 9.32 27.56 3.47 17.17
sdb 87.37 12.70 0.64 0.42 0.00 0.00 0.00 0.00 19.99 1.86 1.60 7.47 33.98 3.04 30.41
sdi 199.00 66.73 1.10 2.15 0.00 0.17 0.00 0.25 9.54 11.96 2.33 5.65 33.06 1.76 46.68
sdj 182.17 47.70 1.12 11.25 0.00 1.10 0.00 2.25 8.71 61.27 4.28 6.30 241.57 2.06 47.35
sdh 139.37 23.90 0.91 0.61 0.00 0.00 0.00 0.00 7.72 13.03 1.25 6.67 26.30 2.21 36.16
sde 91.53 20.10 0.60 0.96 0.00 0.00 0.00 0.00 19.96 14.18 1.91 6.76 48.86 3.22 35.99
sdk 106.27 8.37 0.65 0.23 0.00 0.00 0.00 0.00 10.79 3.19 1.01 6.25 28.51 4.63 53.05
sdl 132.70 15.33 0.70 0.36 0.00 0.00 0.00 0.00 4.93 1.57 0.59 5.43 24.10 1.56 23.12
sda 115.90 16.60 0.67 0.52 0.00 0.00 0.00 0.00 7.45 2.32 0.77 5.88 31.85 1.77 23.49
sdd 103.03 11.23 1.08 0.35 0.00 0.00 0.00 0.00 6.02 2.98 0.54 10.77 31.85 2.54 29.08
cas1-1 207.03 33.50 1.11 0.28 0.00 0.00 0.00 0.00 4.45 2.11 0.99 5.49 8.59 0.86 20.67
cas1-2 401.20 22.57 1.90 0.23 0.00 0.00 0.00 0.00 4.54 34.82 2.61 4.84 10.64 0.70 29.59
cas1-3 197.73 30.17 1.07 0.21 0.00 0.00 0.00 0.00 5.29 4.40 1.18 5.54 7.23 1.35 30.69
cas1-4 199.93 26.33 1.47 0.21 0.00 0.00 0.00 0.00 5.76 7.36 1.35 7.54 8.02 1.30 29.39
cas1-5 529.90 39.10 2.34 0.74 0.00 0.00 0.00 0.00 7.24 25.37 4.83 4.53 19.50 0.66 37.69
cas1-6 128.97 32.07 0.72 0.24 0.00 0.00 0.00 0.00 6.71 148.30 5.62 5.69 7.51 0.91 14.68
cas1-7 171.37 165.23 10.10 13.07 0.00 0.00 0.00 0.00 16.87 727.03 123.02 60.35 81.02 1.70 57.35
cas1-8 221.83 50.47 1.29 0.32 0.00 0.00 0.00 0.00 13.23 10.84 3.48 5.96 6.49 1.30 35.43
cas1-9 466.00 135.57 2.17 1.94 0.00 0.00 0.00 0.00 4.89 150.11 22.63 4.76 14.69 0.73 44.15
cas1-10 320.13 28.57 1.73 10.15 0.00 0.00 0.00 0.00 15.23 625.89 22.76 5.53 363.89 1.43 50.01
cas1-11 643.00 17.47 2.76 0.11 0.00 0.00 0.00 0.00 2.36 11.89 1.73 4.40 6.50 0.92 60.56
cas1-12 207.50 29.90 1.01 0.16 0.00 0.00 0.00 0.00 3.51 5.95 0.91 4.97 5.64 0.90 21.29
12/06/22 11:55:33
avg-cpu: %user %nice %system %iowait %steal %idle
35.50 0.00 10.82 26.01 0.00 27.67
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 995.70 1068.93 15.27 4.27 1940.60 21.43 66.09 1.97 0.21 0.03 0.00 15.70 4.09 0.29 59.05
nvme1n1 0.23 41.40 0.00 0.37 0.00 4.70 0.00 10.20 3.43 0.25 0.01 13.14 9.26 0.15 0.61
dm-0 0.23 46.00 0.00 0.38 0.00 0.00 0.00 0.00 3.43 0.89 0.04 13.14 8.35 0.13 0.61
sdc 52.57 8.73 0.79 0.23 0.00 0.00 0.00 0.00 13.03 2.53 0.61 15.36 26.95 3.58 21.96
sdg 195.30 44.97 10.26 4.99 0.00 0.00 0.00 0.00 12.59 49.40 4.36 53.77 113.55 2.70 64.84
sdf 172.47 10.20 0.90 0.37 0.00 0.00 0.00 0.00 9.13 5.19 1.43 5.36 37.35 1.87 34.15
sdb 58.10 11.97 0.42 0.37 0.00 0.00 0.00 0.00 12.03 1.10 0.59 7.36 31.43 3.22 22.57
sdi 52.33 44.93 0.43 5.32 0.00 0.00 0.00 0.00 14.47 59.78 3.28 8.41 121.33 3.04 29.60
sdj 60.43 73.37 0.60 7.56 0.00 0.17 0.00 0.23 15.98 59.15 5.07 10.22 105.58 2.27 30.36
sdh 104.63 10.87 0.75 0.30 0.00 0.00 0.00 0.00 8.96 3.94 0.87 7.32 28.44 2.37 27.37
sde 97.77 13.73 0.70 0.38 0.00 0.00 0.00 0.00 16.76 2.06 1.46 7.29 28.21 5.41 60.35
sdk 309.37 12.63 1.63 0.33 0.00 0.00 0.00 0.00 24.33 2.26 7.15 5.39 27.11 2.78 89.39
sdl 35.63 10.57 0.39 0.37 0.00 0.00 0.00 0.00 15.46 2.91 0.50 11.15 35.91 4.05 18.69
sda 184.80 64.67 5.10 2.08 0.00 0.00 0.00 0.00 8.93 19.09 2.61 28.26 33.00 2.00 49.95
sdd 75.57 13.83 1.21 0.47 0.00 0.00 0.00 0.00 13.30 3.25 0.89 16.33 35.13 3.11 27.81
cas1-1 327.87 164.57 5.75 1.94 0.00 0.00 0.00 0.00 5.55 62.50 12.11 17.96 12.05 1.03 50.96
cas1-2 181.27 26.87 0.92 0.22 0.00 0.00 0.00 0.00 8.13 1.12 1.50 5.22 8.45 1.18 24.51
cas1-3 173.53 19.10 1.29 0.12 0.00 0.00 0.00 0.00 5.84 1.54 1.04 7.59 6.49 1.21 23.36
cas1-4 404.40 31.63 2.52 0.31 0.00 0.00 0.00 0.00 4.83 1.91 2.01 6.39 9.93 0.75 32.87
cas1-5 250.13 37.57 1.31 0.23 0.00 0.00 0.00 0.00 11.19 1.38 2.85 5.37 6.22 2.32 66.75
cas1-6 411.33 23.23 1.86 0.27 0.00 0.00 0.00 0.00 4.12 3.58 1.78 4.64 11.89 0.80 34.84
cas1-7 323.97 62.47 10.80 4.86 0.00 0.00 0.00 0.00 8.92 265.68 19.49 34.15 79.75 1.82 70.41
cas1-8 250.53 27.87 1.33 0.20 0.00 0.00 0.00 0.00 4.57 2.68 1.22 5.45 7.31 1.17 32.71
cas1-9 163.83 70.03 0.96 5.26 0.00 0.00 0.00 0.00 5.47 277.98 20.36 5.98 76.93 1.47 34.32
cas1-10 204.37 136.13 1.19 7.46 0.00 0.00 0.00 0.00 6.13 463.18 64.31 5.96 56.08 0.85 28.96
cas1-11 1194.00 32.93 5.14 0.18 0.00 0.00 0.00 0.00 7.70 3.08 9.29 4.40 5.67 0.76 93.80
cas1-12 149.43 23.97 0.84 0.23 0.00 0.00 0.00 0.00 3.88 5.50 0.71 5.76 9.95 1.21 20.97
12/06/22 11:56:03
avg-cpu: %user %nice %system %iowait %steal %idle
34.76 0.00 10.64 26.56 0.00 28.03
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 912.07 1367.63 14.47 5.45 1630.07 19.97 64.12 1.44 0.20 0.03 0.00 16.25 4.08 0.25 57.33
nvme1n1 0.23 63.73 0.00 0.59 0.00 3.97 0.00 5.86 4.71 0.23 0.01 13.14 9.41 0.10 0.67
dm-0 0.23 67.53 0.00 0.59 0.00 0.00 0.00 0.00 4.57 0.58 0.04 13.14 8.89 0.10 0.67
sdc 57.83 11.73 0.45 0.34 0.00 0.00 0.00 0.00 17.89 12.90 1.06 8.03 29.85 3.32 23.11
sdg 152.60 11.53 8.32 0.32 0.00 0.00 0.00 0.00 11.85 4.28 1.65 55.85 28.72 3.46 56.79
sdf 103.57 13.27 0.72 0.59 0.00 0.00 0.00 0.00 12.51 11.63 1.30 7.11 45.32 3.31 38.69
sdb 45.60 12.47 0.40 0.34 0.00 0.00 0.00 0.00 12.52 1.61 0.48 9.08 28.26 3.95 22.92
sdi 69.53 29.47 0.52 2.68 0.00 0.10 0.00 0.34 15.92 22.28 1.60 7.72 93.15 2.72 26.96
sdj 194.30 51.73 1.05 4.85 0.00 0.57 0.00 1.08 9.81 49.25 4.19 5.52 95.99 1.99 48.91
sdh 73.77 12.33 0.87 0.33 0.00 0.00 0.00 0.00 14.80 3.03 0.98 12.07 27.69 2.91 25.03
sde 178.10 15.03 0.96 0.37 0.00 0.00 0.00 0.00 11.19 3.76 1.87 5.55 25.31 2.14 41.35
sdk 191.50 108.50 0.98 5.33 0.00 0.53 0.00 0.49 22.00 41.44 8.29 5.24 50.27 2.63 78.91
sdl 60.57 101.97 0.49 5.69 0.00 0.07 0.00 0.07 18.26 35.82 4.46 8.33 57.17 1.88 30.61
sda 121.47 134.27 8.49 9.11 0.00 0.47 0.00 0.35 16.90 42.21 7.41 71.61 69.50 1.72 43.89
sdd 60.00 9.07 0.48 0.23 0.00 0.00 0.00 0.00 14.13 11.53 0.83 8.25 25.97 3.27 22.57
cas1-1 281.97 198.50 9.18 14.32 0.00 0.00 0.00 0.00 10.24 684.94 138.85 33.34 73.89 0.91 43.81
cas1-2 198.30 29.53 1.03 0.21 0.00 0.00 0.00 0.00 3.00 1.54 0.64 5.32 7.14 0.90 20.56
cas1-3 214.30 28.33 1.09 0.21 0.00 0.00 0.00 0.00 10.55 1.10 2.29 5.21 7.67 1.15 27.79
cas1-4 260.70 21.30 1.32 0.11 0.00 0.00 0.00 0.00 3.74 97.58 3.05 5.20 5.20 0.99 27.79
cas1-5 342.73 36.00 1.62 0.20 0.00 0.00 0.00 0.00 9.99 9.13 3.75 4.85 5.82 1.31 49.55
cas1-6 314.83 35.63 1.58 0.47 0.00 0.00 0.00 0.00 9.30 49.86 4.71 5.15 13.55 1.17 40.93
cas1-7 465.50 27.23 9.59 0.19 0.00 0.00 0.00 0.00 9.23 2.37 4.36 21.10 7.07 1.29 63.52
cas1-8 253.00 30.77 1.68 0.20 0.00 0.00 0.00 0.00 5.63 6.11 1.61 6.79 6.76 0.99 28.11
cas1-9 218.77 49.67 1.21 2.56 0.00 0.00 0.00 0.00 5.74 377.14 19.99 5.66 52.82 1.11 29.77
cas1-10 383.30 77.30 1.82 4.76 0.00 0.00 0.00 0.00 12.58 247.93 23.99 4.85 63.00 1.15 53.13
cas1-11 318.30 214.97 1.49 5.17 0.00 0.00 0.00 0.00 18.01 510.24 115.42 4.80 24.63 1.56 83.43
cas1-12 204.83 214.17 1.11 5.59 0.00 0.00 0.00 0.00 9.00 234.18 52.00 5.54 26.74 0.76 31.69
12/06/22 11:56:33
avg-cpu: %user %nice %system %iowait %steal %idle
33.06 0.00 10.62 26.69 0.00 29.62
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 754.47 1330.17 13.01 5.32 1374.73 28.90 64.57 2.13 0.21 0.04 0.00 17.65 4.10 0.25 53.09
nvme1n1 0.20 63.47 0.00 0.49 0.00 4.27 0.00 6.30 6.83 0.23 0.01 9.33 7.95 0.11 0.68
dm-0 0.20 67.67 0.00 0.49 0.00 0.00 0.00 0.00 6.67 0.57 0.04 9.33 7.46 0.10 0.68
sdc 62.07 15.03 0.57 0.43 0.00 0.00 0.00 0.00 15.71 3.69 0.89 9.41 29.48 3.69 28.43
sdg 125.77 14.50 4.93 0.42 0.00 0.00 0.00 0.00 20.46 2.76 2.41 40.16 29.61 4.33 60.73
sdf 51.90 15.33 0.46 0.33 0.00 0.00 0.00 0.00 13.29 6.88 0.68 9.11 22.01 3.32 22.35
sdb 67.90 11.40 0.59 0.32 0.00 0.00 0.00 0.00 13.33 1.96 0.79 8.89 29.12 3.01 23.91
sdi 82.93 16.33 0.58 0.40 0.00 0.00 0.00 0.00 23.81 4.59 1.87 7.10 25.00 2.86 28.36
sdj 86.23 58.83 0.62 7.73 0.00 0.57 0.00 0.95 22.23 63.79 5.46 7.40 134.50 3.13 45.37
sdh 48.57 8.77 0.40 0.33 0.00 0.00 0.00 0.00 13.86 1.44 0.58 8.34 38.28 3.43 19.65
sde 148.63 63.63 1.17 2.01 0.00 0.80 0.00 1.24 22.05 23.31 4.37 8.07 32.41 1.47 31.11
sdk 167.83 110.10 0.98 10.88 0.00 2.03 0.00 1.81 22.15 28.53 6.47 5.99 101.17 2.55 71.00
sdl 43.50 126.30 0.40 12.51 0.00 1.37 0.00 1.07 23.15 52.50 7.34 9.43 101.44 2.40 40.79
sda 221.13 63.23 7.71 13.64 0.00 2.80 0.00 4.24 8.67 62.08 5.56 35.72 220.89 1.70 48.25
sdd 56.53 9.60 0.56 0.27 0.00 0.00 0.00 0.00 16.98 1.50 0.85 10.10 28.88 3.26 21.53
cas1-1 341.07 80.33 8.28 8.11 0.00 0.00 0.00 0.00 10.15 285.13 26.37 24.86 103.36 1.30 54.64
cas1-2 226.10 27.37 1.29 0.21 0.00 0.00 0.00 0.00 6.67 4.40 1.63 5.84 7.83 1.34 33.91
cas1-3 201.47 33.87 1.16 0.26 0.00 0.00 0.00 0.00 9.14 70.88 4.24 5.87 7.77 1.75 41.08
cas1-4 159.87 25.30 1.00 0.17 0.00 0.00 0.00 0.00 9.85 15.07 1.96 6.41 7.03 1.61 29.83
cas1-5 302.83 205.70 1.80 2.15 0.00 0.00 0.00 0.00 12.39 98.45 24.00 6.09 10.69 0.74 37.85
cas1-6 167.87 32.27 0.94 0.16 0.00 0.00 0.00 0.00 4.71 9.26 1.09 5.75 5.07 1.17 23.41
cas1-7 390.43 32.83 6.19 0.22 0.00 0.00 0.00 0.00 10.97 5.74 4.47 16.22 6.93 1.66 70.13
cas1-8 155.63 19.17 0.84 0.21 0.00 0.00 0.00 0.00 11.05 4.92 1.81 5.51 11.47 1.18 20.56
cas1-9 214.30 32.37 1.20 0.18 0.00 0.00 0.00 0.00 12.06 5.10 2.75 5.72 5.72 1.26 31.07
cas1-10 229.47 82.43 1.23 7.59 0.00 0.00 0.00 0.00 20.07 430.06 40.06 5.48 94.32 1.65 51.43
cas1-11 367.30 159.90 1.82 10.67 0.00 0.00 0.00 0.00 14.80 524.93 89.37 5.06 68.31 1.39 73.43
cas1-12 172.00 176.60 1.00 12.36 0.00 0.00 0.00 0.00 7.82 815.75 145.41 5.93 71.64 1.22 42.63
12/06/22 11:57:03
avg-cpu: %user %nice %system %iowait %steal %idle
29.45 0.00 9.74 25.08 0.00 35.73
Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 693.90 1355.13 11.20 5.41 878.27 24.53 55.86 1.78 0.19 0.04 0.00 16.52 4.09 0.24 48.37
nvme1n1 0.03 59.00 0.00 0.48 0.00 3.90 0.00 6.20 8.00 0.26 0.01 16.00 8.31 0.12 0.71
dm-0 0.03 62.80 0.00 0.48 0.00 0.00 0.00 0.00 4.00 0.67 0.04 16.00 7.82 0.11 0.71
sdc 303.63 54.97 3.34 1.82 0.00 0.33 0.00 0.60 16.44 21.18 5.43 11.27 33.94 0.94 33.84
sdg 83.93 8.80 3.00 0.23 0.00 0.00 0.00 0.00 15.91 3.11 1.24 36.65 27.06 4.06 37.65
sdf 37.63 1.83 0.40 0.13 0.00 0.00 0.00 0.00 11.66 1.13 0.38 10.75 72.15 2.88 11.35
sdb 85.53 8.30 0.53 0.23 0.00 0.00 0.00 0.00 9.64 2.24 0.71 6.34 28.92 2.25 21.09
sdi 30.00 5.13 0.34 0.12 0.00 0.00 0.00 0.00 12.26 1.95 0.31 11.56 24.75 3.72 13.05
sdj 28.43 10.20 0.25 0.23 0.00 0.00 0.00 0.00 14.97 1.34 0.38 8.89 23.56 3.75 14.49
sdh 34.60 8.97 0.25 0.27 0.00 0.00 0.00 0.00 12.43 1.09 0.36 7.36 30.38 3.34 14.56
sde 197.50 262.73 0.84 10.14 0.00 0.37 0.00 0.14 32.98 58.69 21.21 4.35 39.54 1.84 84.56
sdk 77.33 59.13 0.56 6.84 0.00 0.23 0.00 0.39 15.24 40.62 3.36 7.36 118.45 2.82 38.45
sdl 27.07 33.57 0.22 2.46 0.00 0.00 0.00 0.00 14.54 11.18 0.67 8.51 74.95 2.70 16.39
sda 68.20 20.47 4.32 1.56 0.00 0.00 0.00 0.00 9.48 119.63 2.98 64.80 78.22 2.51 22.24
sdd 32.33 3.30 0.35 0.11 0.00 0.00 0.00 0.00 11.66 2.59 0.33 10.96 33.49 3.13 11.15
cas1-1 157.10 28.63 4.69 1.47 0.00 0.00 0.00 0.00 15.75 3715.18 108.85 30.55 52.70 2.61 48.51
cas1-2 225.37 19.73 1.08 0.15 0.00 0.00 0.00 0.00 49.25 34.03 11.77 4.92 7.92 2.09 51.15
cas1-3 387.20 122.10 3.69 1.55 0.00 0.00 0.00 0.00 20.66 85.34 18.42 9.76 12.98 0.69 34.99
cas1-4 139.77 15.57 0.77 0.12 0.00 0.00 0.00 0.00 3.84 0.96 0.55 5.65 7.98 0.96 14.88
cas1-5 315.90 464.30 1.36 9.94 0.00 0.00 0.00 0.00 20.04 1709.14 799.88 4.41 21.93 1.03 80.41
cas1-6 133.20 9.80 0.81 0.12 0.00 0.00 0.00 0.00 3.76 3.76 0.54 6.25 13.06 1.02 14.60
cas1-7 210.77 25.27 3.61 0.14 0.00 0.00 0.00 0.00 14.51 1.95 3.11 17.55 5.49 2.56 60.44
cas1-8 133.77 24.43 0.66 0.18 0.00 0.00 0.00 0.00 6.96 16.52 1.33 5.07 7.69 1.68 26.65
cas1-9 104.33 9.63 0.66 0.05 0.00 0.00 0.00 0.00 24.99 1.12 2.62 6.50 5.18 4.36 49.73
cas1-10 100.90 22.27 0.55 0.14 0.00 0.00 0.00 0.00 17.45 6.49 1.90 5.58 6.33 2.26 27.83
cas1-11 182.03 96.23 0.99 6.68 0.00 0.00 0.00 0.00 10.34 234.59 24.46 5.57 71.09 1.57 43.59
cas1-12 112.33 46.17 0.62 2.35 0.00 0.00 0.00 0.00 8.34 58.96 3.66 5.62 52.21 1.13 17.85Hello @jhindersson
Thank you for posting the iostat output.
Sure, the
unaligned_iois set to 1 as per module defaults. Which should not pass-through requests even if we miss alignment if i understand correctly?
Correct, 1 is the value we expected.
Would it be possible to provide some additional information? One of the possible reasons for such high pass through rate is that the system may be short on memory. This can be verified by reviewing output of free -h command. We would also like to know about kernel allocator stats which you can get with cat /proc/slabinfo.
Do you have a possibility to run additional experiment on a test node? We would like to exclude that the problem is cleaning policy related by switching it to NOP and verifying that the pass through requests statistic still increases. Please be aware that if you have NOP cleaning policy, the pass through requests shouldn't occur only as long as the cache is not 100% dirty. If the cache gets 100% dirty then pass through requests are expected.
Next thing you could try is checking if the problem reproduces on CAS v22.6.1 which is our latest release. Unfortunately, we don't have any upgrade mechanism, so if you would like to do it, it would require reinstall CAS and restore the configuration manually.
BTW would you mind sharing with us the name of application that is running on the top on CAS?
Best Regards, Michal
jhindersson
commented
1 year ago Hi,
During the holidays I have removed 3 systems out of service to make some changes, they now run version v22.6.2 which was the latest stable last week. I have also played around a little bit with different settings to get a feeling of what might happen.
From this experience I have noticed that everything works like a charm as long as there is Free blocks in the cache. As soon as the cache is full things slows down considerably, if i disable an ioclass that contains data everything works normally again until all cache lines have been reused in "active" ioclasses. This happens even though all lines are marked clean which causes me to make an assumption that this might be related to how eviction works (or does not work) when the cache is full.
I believe that there should be sufficient memory available, we run with vm.min_free_kbytes=4G to ensure that there is always free memory for buffers.
This is the free output from one of the smaller machines where i have just installed the latest 22.6.2 release.
# free -h
total used free shared buff/cache available
Mem: 62G 10G 6.1G 1.4M 46G 36G
Swap: 0B 0B 0Bslabinfo:
# cat /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
sw_flow 0 0 1760 18 8 : tunables 0 0 0 : slabdata 0 0 0
nf_conncount_rb 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
Block_Device 16422 16422 176 46 2 : tunables 0 0 0 : slabdata 357 357 0
cas_biovec_128 4320 4320 2136 15 8 : tunables 0 0 0 : slabdata 288 288 0
cas_biovec_64 4176 4176 1112 29 8 : tunables 0 0 0 : slabdata 144 144 0
cas_biovec_32 4320 4320 600 27 4 : tunables 0 0 0 : slabdata 160 160 0
cas_biovec_4 4160 4160 152 26 1 : tunables 0 0 0 : slabdata 160 160 0
cas_biovec_2 4352 4352 120 34 1 : tunables 0 0 0 : slabdata 128 128 0
OCF Composite 4320 4320 272 30 2 : tunables 0 0 0 : slabdata 144 144 0
ocf_mio_64 256 256 27224 1 8 : tunables 0 0 0 : slabdata 256 256 0
ocf_mio_32 256 256 13656 2 8 : tunables 0 0 0 : slabdata 128 128 0
ocf_mio_16 256 256 6872 4 8 : tunables 0 0 0 : slabdata 64 64 0
kvm_vcpu 0 0 17152 1 8 : tunables 0 0 0 : slabdata 0 0 0
ocf_mio_8 4176 4176 3480 9 8 : tunables 0 0 0 : slabdata 464 464 0
kvm_mmu_page_header 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0
x86_fpu 0 0 4160 7 8 : tunables 0 0 0 : slabdata 0 0 0
ocf_mio_4 4320 4320 1784 18 8 : tunables 0 0 0 : slabdata 240 240 0
ocf_mio_2 4480 4480 936 35 8 : tunables 0 0 0 : slabdata 128 128 0
ocf_req_128 4176 4176 3568 9 8 : tunables 0 0 0 : slabdata 464 464 0
ocf_req_64 4096 4096 1968 16 8 : tunables 0 0 0 : slabdata 256 256 0
ocf_req_32 6360 6468 1168 28 8 : tunables 0 0 0 : slabdata 231 231 0
ocf_req_16 4704 4704 768 42 8 : tunables 0 0 0 : slabdata 112 112 0
ocf_req_8 4480 4480 568 28 4 : tunables 0 0 0 : slabdata 160 160 0
ocf_req_4 4352 4352 472 34 4 : tunables 0 0 0 : slabdata 128 128 0
ocf_req_2 4256 4256 424 38 4 : tunables 0 0 0 : slabdata 112 112 0
ocf_req_1 10760 10760 400 40 4 : tunables 0 0 0 : slabdata 269 269 0
casdsk_disk 405 405 360 45 4 : tunables 0 0 0 : slabdata 9 9 0
casdsk_exp_obj 60 60 136 30 1 : tunables 0 0 0 : slabdata 2 2 0
nf_conntrack 57993 58050 320 25 2 : tunables 0 0 0 : slabdata 2322 2322 0
ext4_groupinfo_4k 1434160 1434160 144 28 1 : tunables 0 0 0 : slabdata 51220 51220 0
btrfs_delayed_node 0 0 312 26 2 : tunables 0 0 0 : slabdata 0 0 0
btrfs_ordered_extent 0 0 416 39 4 : tunables 0 0 0 : slabdata 0 0 0
btrfs_inode 0 0 1168 28 8 : tunables 0 0 0 : slabdata 0 0 0
scsi_sense_cache 5088 5088 128 32 1 : tunables 0 0 0 : slabdata 159 159 0
fsverity_info 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0
ip6-frags 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
PINGv6 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0
RAWv6 52 52 1216 26 8 : tunables 0 0 0 : slabdata 2 2 0
UDPv6 1512 1512 1344 24 8 : tunables 0 0 0 : slabdata 63 63 0
tw_sock_TCPv6 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0
request_sock_TCPv6 0 0 304 26 2 : tunables 0 0 0 : slabdata 0 0 0
TCPv6 520 520 2432 13 8 : tunables 0 0 0 : slabdata 40 40 0
kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 : slabdata 0 0 0
dm_uevent 0 0 2632 12 8 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
fuse_request 4480 4480 144 28 1 : tunables 0 0 0 : slabdata 160 160 0
fuse_inode 39 39 832 39 8 : tunables 0 0 0 : slabdata 1 1 0
ecryptfs_key_record_cache 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0
ecryptfs_inode_cache 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
ecryptfs_file_cache 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
ecryptfs_auth_tok_list_item 0 0 832 39 8 : tunables 0 0 0 : slabdata 0 0 0
fat_inode_cache 176 176 744 44 8 : tunables 0 0 0 : slabdata 4 4 0
fat_cache 102 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0
squashfs_inode_cache 46 46 704 46 8 : tunables 0 0 0 : slabdata 1 1 0
jbd2_journal_handle 1360 1360 48 85 1 : tunables 0 0 0 : slabdata 16 16 0
jbd2_journal_head 6009 6086 120 34 1 : tunables 0 0 0 : slabdata 179 179 0
jbd2_revoke_table_s 2560 2560 16 256 1 : tunables 0 0 0 : slabdata 10 10 0
ext4_inode_cache 14978073 15718077 1096 29 8 : tunables 0 0 0 : slabdata 550467 550467 0
ext4_allocation_context 512 512 128 32 1 : tunables 0 0 0 : slabdata 16 16 0
ext4_system_zone 90780 90780 40 102 1 : tunables 0 0 0 : slabdata 890 890 0
ext4_io_end 15232 15232 64 64 1 : tunables 0 0 0 : slabdata 238 238 0
ext4_pending_reservation 2048 2048 32 128 1 : tunables 0 0 0 : slabdata 16 16 0
ext4_extent_status 68926 95880 40 102 1 : tunables 0 0 0 : slabdata 940 940 0
mbcache 26323 28981 56 73 1 : tunables 0 0 0 : slabdata 397 397 0
userfaultfd_ctx_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
dnotify_struct 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
pid_namespace 0 0 208 39 2 : tunables 0 0 0 : slabdata 0 0 0
ip4-frags 0 0 200 40 2 : tunables 0 0 0 : slabdata 0 0 0
xfrm_state 0 0 704 46 8 : tunables 0 0 0 : slabdata 0 0 0
PING 9060 9060 1024 32 8 : tunables 0 0 0 : slabdata 284 284 0
RAW 32 32 1024 32 8 : tunables 0 0 0 : slabdata 1 1 0
tw_sock_TCP 19930 20757 248 33 2 : tunables 0 0 0 : slabdata 629 629 0
request_sock_TCP 416 416 304 26 2 : tunables 0 0 0 : slabdata 16 16 0
TCP 2856 2856 2240 14 8 : tunables 0 0 0 : slabdata 204 204 0
hugetlbfs_inode_cache 50 50 632 25 4 : tunables 0 0 0 : slabdata 2 2 0
dquot 2464 2464 256 32 2 : tunables 0 0 0 : slabdata 77 77 0
eventpoll_pwq 10696 10696 72 56 1 : tunables 0 0 0 : slabdata 191 191 0
dax_cache 84 84 768 42 8 : tunables 0 0 0 : slabdata 2 2 0
request_queue 165 165 2104 15 8 : tunables 0 0 0 : slabdata 11 11 0
biovec-max 416 416 4096 8 8 : tunables 0 0 0 : slabdata 52 52 0
biovec-128 1456 1520 2048 16 8 : tunables 0 0 0 : slabdata 95 95 0
biovec-64 3232 3232 1024 32 8 : tunables 0 0 0 : slabdata 101 101 0
khugepaged_mm_slot 36 36 112 36 1 : tunables 0 0 0 : slabdata 1 1 0
user_namespace 0 0 536 30 4 : tunables 0 0 0 : slabdata 0 0 0
uid_cache 10610 11328 128 32 1 : tunables 0 0 0 : slabdata 354 354 0
dmaengine-unmap-256 15 15 2112 15 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-128 30 30 1088 30 8 : tunables 0 0 0 : slabdata 1 1 0
sock_inode_cache 19350 19426 832 39 8 : tunables 0 0 0 : slabdata 499 499 0
skbuff_ext_cache 9030 9030 192 42 2 : tunables 0 0 0 : slabdata 215 215 0
skbuff_fclone_cache 9216 9280 512 32 4 : tunables 0 0 0 : slabdata 290 290 0
skbuff_head_cache 22048 22048 256 32 2 : tunables 0 0 0 : slabdata 689 689 0
file_lock_cache 4736 4736 216 37 2 : tunables 0 0 0 : slabdata 128 128 0
fsnotify_mark_connector 2176 2176 32 128 1 : tunables 0 0 0 : slabdata 17 17 0
net_namespace 0 0 4928 6 8 : tunables 0 0 0 : slabdata 0 0 0
task_delay_info 17850 17850 80 51 1 : tunables 0 0 0 : slabdata 350 350 0
taskstats 4512 4512 344 47 4 : tunables 0 0 0 : slabdata 96 96 0
proc_dir_entry 1680 1680 192 42 2 : tunables 0 0 0 : slabdata 40 40 0
pde_opener 18564 18564 40 102 1 : tunables 0 0 0 : slabdata 182 182 0
proc_inode_cache 19439 20736 680 24 4 : tunables 0 0 0 : slabdata 864 864 0
bdev_cache 1092 1092 832 39 8 : tunables 0 0 0 : slabdata 28 28 0
shmem_inode_cache 7895 8145 720 45 8 : tunables 0 0 0 : slabdata 181 181 0
kernfs_node_cache 120738 121410 136 30 1 : tunables 0 0 0 : slabdata 4047 4047 0
mnt_cache 4475 4475 320 25 2 : tunables 0 0 0 : slabdata 179 179 0
filp 93956 95744 256 32 2 : tunables 0 0 0 : slabdata 2992 2992 0
inode_cache 43991 46228 608 26 4 : tunables 0 0 0 : slabdata 1778 1778 0
dentry 12501974 13367130 192 42 2 : tunables 0 0 0 : slabdata 318265 318265 0
names_cache 224 224 4096 8 8 : tunables 0 0 0 : slabdata 28 28 0
iint_cache 0 0 120 34 1 : tunables 0 0 0 : slabdata 0 0 0
lsm_file_cache 5780 5780 24 170 1 : tunables 0 0 0 : slabdata 34 34 0
buffer_head 5552798 6299826 104 39 1 : tunables 0 0 0 : slabdata 161534 161534 0
uts_namespace 0 0 440 37 4 : tunables 0 0 0 : slabdata 0 0 0
nsproxy 100059 103076 56 73 1 : tunables 0 0 0 : slabdata 1412 1412 0
vm_area_struct 117429 117975 208 39 2 : tunables 0 0 0 : slabdata 3025 3025 0
mm_struct 9060 9060 1088 30 8 : tunables 0 0 0 : slabdata 302 302 0
files_cache 10810 10810 704 46 8 : tunables 0 0 0 : slabdata 235 235 0
signal_cache 7980 7980 1152 28 8 : tunables 0 0 0 : slabdata 285 285 0
sighand_cache 4650 4680 2112 15 8 : tunables 0 0 0 : slabdata 312 312 0
task_struct 2971 3165 6016 5 8 : tunables 0 0 0 : slabdata 633 633 0
cred_jar 21798 21798 192 42 2 : tunables 0 0 0 : slabdata 519 519 0
anon_vma_chain 175127 175680 64 64 1 : tunables 0 0 0 : slabdata 2745 2745 0
anon_vma 97996 98475 104 39 1 : tunables 0 0 0 : slabdata 2525 2525 0
pid 32608 32608 128 32 1 : tunables 0 0 0 : slabdata 1019 1019 0
Acpi-Operand 11536 11536 72 56 1 : tunables 0 0 0 : slabdata 206 206 0
Acpi-ParseExt 20904 20904 104 39 1 : tunables 0 0 0 : slabdata 536 536 0
Acpi-State 1071 1071 80 51 1 : tunables 0 0 0 : slabdata 21 21 0
numa_policy 31 31 264 31 2 : tunables 0 0 0 : slabdata 1 1 0
trace_event_file 2024 2024 88 46 1 : tunables 0 0 0 : slabdata 44 44 0
ftrace_event_field 10710 10710 48 85 1 : tunables 0 0 0 : slabdata 126 126 0
pool_workqueue 2770 3136 256 32 2 : tunables 0 0 0 : slabdata 98 98 0
radix_tree_node 1855590 2188144 584 28 4 : tunables 0 0 0 : slabdata 78148 78148 0
task_group 425 425 640 25 4 : tunables 0 0 0 : slabdata 17 17 0
vmap_area 3189285 5343360 64 64 1 : tunables 0 0 0 : slabdata 83490 83490 0
dma-kmalloc-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-192 588 588 192 42 2 : tunables 0 0 0 : slabdata 14 14 0
kmalloc-rcl-128 2268 2592 128 32 1 : tunables 0 0 0 : slabdata 81 81 0
kmalloc-rcl-96 6528 6846 96 42 1 : tunables 0 0 0 : slabdata 163 163 0
kmalloc-rcl-64 3809301 4062016 64 64 1 : tunables 0 0 0 : slabdata 63469 63469 0
kmalloc-rcl-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-8k 1080 1084 8192 4 8 : tunables 0 0 0 : slabdata 271 271 0
kmalloc-4k 3962 3984 4096 8 8 : tunables 0 0 0 : slabdata 498 498 0
kmalloc-2k 4910 4992 2048 16 8 : tunables 0 0 0 : slabdata 312 312 0
kmalloc-1k 15575 15648 1024 32 8 : tunables 0 0 0 : slabdata 489 489 0
kmalloc-512 47954 48032 512 32 4 : tunables 0 0 0 : slabdata 1501 1501 0
kmalloc-256 4384 4384 256 32 2 : tunables 0 0 0 : slabdata 137 137 0
kmalloc-192 12096 12096 192 42 2 : tunables 0 0 0 : slabdata 288 288 0
kmalloc-128 63348 63392 128 32 1 : tunables 0 0 0 : slabdata 1981 1981 0
kmalloc-96 10248 10248 96 42 1 : tunables 0 0 0 : slabdata 244 244 0
kmalloc-64 57705 59776 64 64 1 : tunables 0 0 0 : slabdata 934 934 0
kmalloc-32 80896 80896 32 128 1 : tunables 0 0 0 : slabdata 632 632 0
kmalloc-16 136192 136192 16 256 1 : tunables 0 0 0 : slabdata 532 532 0
kmalloc-8 51712 51712 8 512 1 : tunables 0 0 0 : slabdata 101 101 0
kmem_cache_node 2496 2496 64 64 1 : tunables 0 0 0 : slabdata 39 39 0
kmem_cache 2376 2376 448 36 4 : tunables 0 0 0 : slabdata 66 66 0I can give the NOP policy a try to see if it changes anything.
This is a OpenStack Swift object storage setup which relies heavily on the cache to handle the load. Data transfer throughput and latency is mostly not an issue but metadata for hundreds of millions of files are and is where we need the cache most.
Right now i do run with a ioclass config as follows, which seems to work mostly fine until the unclassified class fills up the cache completely and pass-through is observed on all classes. I would assume that cache lines from the unclassified class would be evicted if metadata is read/written but that does not seem to be the case?
IO class id,IO class name,Eviction priority,Allocation
0,unclassified,22,1
1,metadata&done,,1
3,file_name_prefix:hashes&done,3,1
8,file_size:le:262144&done,8,1
22,direct&done,20,1I have also changed to ALRU cleaning with the config casadm -X -n cleaning-alru -i 1 -t 0 -w 1 -b 1000 which keeps the cache clean even during high activity.
Best Regards, Joakim
Question
I have a question wondering what causes high rate of Pass Through Reads/Writes when in Write Back mode. This happens on all io-classes but bothers me most on the metadata class where we need the cache most.
We are serving millions of small files where the metadata offload from HDD is the most important part where the cache makes most impact.
So what I really try to understand is how and why some requests is put in pass through, cache is always 100% occupied and almost always fully clean.
Motivation
We are facing performance issues when the cache is not utilized for all requests.
Your Environment
OpenCAS version (commit hash or tag): 21.6.3 and 21.6.4
Operating System: Ubuntu 18.04 with HWE kernel
Kernel version: 5.4.0-104-generic
Cache device type (NAND/Optane/other): NAND Samsung NVMe 3.8TB
Core device type (HDD/SSD/other): HDD of different sizes, mostly 16TB 7,2KRPM SAS disks
Cache configuration:
Other (e.g. lsblk, casadm -P, casadm -L)
casadm -L
casadm -P -i 1 (cutoff error stats which is 0)
╔══════════════════╤═══════════╤═══════╤═════════════╗ ║ Usage statistics │ Count │ % │ Units ║ ╠══════════════════╪═══════════╪═══════╪═════════════╣ ║ Occupancy │ 883706592 │ 100.0 │ 4KiB Blocks ║ ║ Free │ 0 │ 0.0 │ 4KiB Blocks ║ ║ Clean │ 883701336 │ 100.0 │ 4KiB Blocks ║ ║ Dirty │ 5256 │ 0.0 │ 4KiB Blocks ║ ╚══════════════════╧═══════════╧═══════╧═════════════╝
╔══════════════════════╤═══════════╤═══════╤══════════╗ ║ Request statistics │ Count │ % │ Units ║ ╠══════════════════════╪═══════════╪═══════╪══════════╣ ║ Read hits │ 201485232 │ 40.7 │ Requests ║ ║ Read partial misses │ 9 │ 0.0 │ Requests ║ ║ Read full misses │ 2676256 │ 0.5 │ Requests ║ ║ Read total │ 204161497 │ 41.2 │ Requests ║ ╟──────────────────────┼───────────┼───────┼──────────╢ ║ Write hits │ 69994713 │ 14.1 │ Requests ║ ║ Write partial misses │ 3270 │ 0.0 │ Requests ║ ║ Write full misses │ 855078 │ 0.2 │ Requests ║ ║ Write total │ 70853061 │ 14.3 │ Requests ║ ╟──────────────────────┼───────────┼───────┼──────────╢ ║ Pass-Through reads │ 77532129 │ 15.7 │ Requests ║ ║ Pass-Through writes │ 142661599 │ 28.8 │ Requests ║ ║ Serviced requests │ 275014558 │ 55.5 │ Requests ║ ╟──────────────────────┼───────────┼───────┼──────────╢ ║ Total requests │ 495208286 │ 100.0 │ Requests ║ ╚══════════════════════╧═══════════╧═══════╧══════════╝
╔══════════════════════════════════╤════════════╤═══════╤═════════════╗ ║ Block statistics │ Count │ % │ Units ║ ╠══════════════════════════════════╪════════════╪═══════╪═════════════╣ ║ Reads from core(s) │ 1131293796 │ 51.2 │ 4KiB Blocks ║ ║ Writes to core(s) │ 1078257685 │ 48.8 │ 4KiB Blocks ║ ║ Total to/from core(s) │ 2209551481 │ 100.0 │ 4KiB Blocks ║ ╟──────────────────────────────────┼────────────┼───────┼─────────────╢ ║ Reads from cache │ 667055407 │ 89.1 │ 4KiB Blocks ║ ║ Writes to cache │ 81529123 │ 10.9 │ 4KiB Blocks ║ ║ Total to/from cache │ 748584530 │ 100.0 │ 4KiB Blocks ║ ╟──────────────────────────────────┼────────────┼───────┼─────────────╢ ║ Reads from exported object(s) │ 1373377362 │ 62.3 │ 4KiB Blocks ║ ║ Writes to exported object(s) │ 831034120 │ 37.7 │ 4KiB Blocks ║ ║ Total to/from exported object(s) │ 2204411482 │ 100.0 │ 4KiB Blocks ║ ╚══════════════════════════════════╧════════════╧═══════╧═════════════╝
$ casadm -P -i 1 -j 1 Core Id 1 Core Device /dev/sdb1 Exported Object /dev/cas1-1 Core Size 3906469627 [4KiB Blocks] / 14902.00 [GiB] Dirty for 7 [s] / 7 [s] Status Active Seq cutoff threshold 1024 [KiB] Seq cutoff policy never