Performance drops when flushing

[allentkw]

Question

Performance seems to take a hit when the cache is cleaning into the core. Using WO mode and ACP flushing

I am not sure whether if this is expected. Wouldn't the writes continued to hit the cache first even though the cache may be cleaning into the core at the same time ?

I am setting fsync in fio as I am trying to test the performance when using the cached device in a ceph environment (I believe ceph uses fsync for its writes?)

For example:

Baseline - fio against the cache device (Updated with the correct fio results for the device)

fsync-blocksize4k-iodepth1-randwrite: (groupid=0, jobs=1): err= 0: pid=5855: Thu Aug 26 21:44:39 2021
  write: IOPS=37.5k, BW=147MiB/s (154MB/s)(8797MiB/60001msec)
fsync-blocksize4k-iodepth32-randwrite: (groupid=1, jobs=1): err= 0: pid=5932: Thu Aug 26 21:44:39 2021
  write: IOPS=146k, BW=571MiB/s (598MB/s)(33.4GiB/60001msec)
fsync-blocksize4k-iodepth128-randwrite: (groupid=2, jobs=1): err= 0: pid=6033: Thu Aug 26 21:44:39 2021
  write: IOPS=145k, BW=568MiB/s (596MB/s)(33.3GiB/60001msec)

Performing simultaneous fio on /dev/cas1-1

fsync-blocksize4k-iodepth1-randwrite: (groupid=0, jobs=1): err= 0: pid=7372: Wed Sep  1 06:36:57 2021
  write: IOPS=934, BW=3739KiB/s (3829kB/s)(219MiB/60022msec)
fsync-blocksize4k-iodepth4-randwrite: (groupid=1, jobs=1): err= 0: pid=7554: Wed Sep  1 06:36:57 2021
  write: IOPS=3660, BW=14.3MiB/s (14.0MB/s)(859MiB/60051msec)
fsync-blocksize4k-iodepth32-randwrite: (groupid=2, jobs=1): err= 0: pid=7722: Wed Sep  1 06:36:57 2021
  write: IOPS=3424, BW=13.4MiB/s (14.0MB/s)(804MiB/60076msec)

Performing simultaneous fio on /dev/cas1-2

fsync-blocksize4k-iodepth1-randwrite: (groupid=0, jobs=1): err= 0: pid=7429: Wed Sep  1 06:36:59 2021
  write: IOPS=3090, BW=12.1MiB/s (12.7MB/s)(724MiB/60001msec)
fsync-blocksize4k-iodepth4-randwrite: (groupid=1, jobs=1): err= 0: pid=7569: Wed Sep  1 06:36:59 2021
  write: IOPS=17.1k, BW=66.7MiB/s (69.0MB/s)(4004MiB/60001msec)
fsync-blocksize4k-iodepth32-randwrite: (groupid=2, jobs=1): err= 0: pid=7737: Wed Sep  1 06:36:59 2021
  write: IOPS=14.4k, BW=56.3MiB/s (59.0MB/s)(3378MiB/60001msec)

Performing simultaneous fio on /dev/cas1-3

fsync-blocksize4k-iodepth1-randwrite: (groupid=0, jobs=1): err= 0: pid=7481: Wed Sep  1 06:37:05 2021
  write: IOPS=11.4k, BW=44.7MiB/s (46.9MB/s)(2682MiB/60001msec)
fsync-blocksize4k-iodepth4-randwrite: (groupid=1, jobs=1): err= 0: pid=7586: Wed Sep  1 06:37:05 2021
  write: IOPS=15.0k, BW=58.6MiB/s (61.5MB/s)(3517MiB/60001msec)
fsync-blocksize4k-iodepth32-randwrite: (groupid=2, jobs=1): err= 0: pid=7756: Wed Sep  1 06:37:05 2021
  write: IOPS=11.7k, BW=45.8MiB/s (48.0MB/s)(2755MiB/60146msec)

Observed iostat (Constant writes into sda)

01/09/2021 06:35:56
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.48    0.00    5.27    2.26    0.00   92.00

Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
nvme0n1        951.00 58250.00      3.71    227.54     0.00     0.00   0.00   0.00    0.03    0.01   0.09     4.00     4.00   0.00   6.40
sda              0.00  984.00      0.00      3.78     0.00     0.00   0.00   0.00    0.00   99.13  94.12     0.00     3.93   0.87  85.60
sdb              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdc              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdd              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdf              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sde              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdg              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdh              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
cas1-1           0.00 7816.00      0.00     15.27     0.00     0.00   0.00   0.00    0.00    0.11   0.00     0.00     2.00   0.00   0.00
cas1-2           0.00 23095.00      0.00     45.11     0.00     0.00   0.00   0.00    0.00    0.06   0.00     0.00     2.00   0.00   0.00
cas1-3           0.00 29744.00      0.00     58.09     0.00     0.00   0.00   0.00    0.00    0.04   0.00     0.00     2.00   0.00   0.00
cas1-4           0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
cas1-5           0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
cas1-6           0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
cas1-7           0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
cas1-8           0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00

Your Environment

• OpenCAS version (commit hash or tag): 20.03.4.0311
• Operating System: Ubuntu 18.04.5 LTS
• Kernel version: 4.15.0-154-generic #161-Ubuntu
• Cache device type (NAND/Optane/other): Intel Optane 900P (SSDPED1D480GA)
• Core device type (HDD/SSD/other): 10k rpm HDD (HUC109090CSS600)
• Cache configuration:
• • Cache mode: wo
• • Cache line size: 4
• • Promotion policy: always
• • Cleaning policy: acp
• • Sequential cutoff policy: (default: full)

fio_jobfile

[global]
ioengine=libaio
runtime=60
time_based
direct=1
group_reporting=1
numjobs=1

[fsync-blocksize4k-iodepth1-randwrite]
stonewall=1
fsync=1
bs=4k
iodepth=1
rw=randwrite

[fsync-blocksize4k-iodepth4-randwrite]
stonewall=1
fsync=1
bs=4k
iodepth=4
rw=randwrite

[fsync-blocksize4k-iodepth32-randwrite]
stonewall=1
fsync=1
bs=4k
iodepth=32
rw=randwrite

lsblk

sda           8:0    0 838.4G  0 disk 
├─sda1        8:1    0 828.4G  0 part 
│ └─cas1-1  252:0    0 828.4G  0 disk 
└─sda2        8:2    0    10G  0 part 
sdb           8:16   0 838.4G  0 disk 
├─sdb1        8:17   0 828.4G  0 part 
│ └─cas1-2  252:1    0 828.4G  0 disk 
└─sdb2        8:18   0    10G  0 part 
sdc           8:32   0 838.4G  0 disk 
├─sdc1        8:33   0 828.4G  0 part 
│ └─cas1-3  252:2    0 828.4G  0 disk 
└─sdc2        8:34   0    10G  0 part 
sdd           8:48   0 838.4G  0 disk 
├─sdd1        8:49   0 828.4G  0 part 
│ └─cas1-4  252:3    0 828.4G  0 disk 
└─sdd2        8:50   0    10G  0 part 
sde           8:64   0 838.4G  0 disk 
├─sde1        8:65   0 828.4G  0 part 
│ └─cas1-5  252:4    0 828.4G  0 disk 
└─sde2        8:66   0    10G  0 part 
sdf           8:80   0 838.4G  0 disk 
├─sdf1        8:81   0 828.4G  0 part 
│ └─cas1-6  252:5    0 828.4G  0 disk 
└─sdf2        8:82   0    10G  0 part 
sdg           8:96   0 838.4G  0 disk 
├─sdg1        8:97   0 828.4G  0 part 
│ └─cas1-7  252:6    0 828.4G  0 disk 
└─sdg2        8:98   0    10G  0 part 
sdh           8:112  0 838.4G  0 disk 
├─sdh1        8:113  0 828.4G  0 part 
│ └─cas1-8  252:7    0 828.4G  0 disk 
└─sdh2        8:114  0    10G  0 part 
nvme0n1     259:0    0 447.1G  0 disk 
├─nvme0n1p1 259:1    0   285M  0 part /boot/efi
├─nvme0n1p2 259:2    0  18.6G  0 part /
└─nvme0n1p3 259:3    0 428.2G  0 part 

casadm -L

# casadm -L
type    id   disk             status    write policy   device
cache   1    /dev/nvme0n1p3   Running   wo             -
+core   1    /dev/sda1        Active    -              /dev/cas1-1
+core   2    /dev/sdb1        Active    -              /dev/cas1-2
+core   3    /dev/sdc1        Active    -              /dev/cas1-3
+core   4    /dev/sdd1        Active    -              /dev/cas1-4
+core   5    /dev/sde1        Active    -              /dev/cas1-5
+core   6    /dev/sdf1        Active    -              /dev/cas1-6
+core   7    /dev/sdg1        Active    -              /dev/cas1-7
+core   8    /dev/sdh1        Active    -              /dev/cas1-8

[mslovy]

I have the similar issue when using writeback mode. do you solve it now?

[arutk]

Hi @allentkw, @mslovy. We'll take a look at this once we have dealt with other open issues. Thanks.