pbench-fio on distributed filesystem: no host-unique filenames

[bengland2]

pbench-fio fails to run a distributed filesystem test in the way that fio --client has been taught to do. What we want to do is specify a test directory, such as /mnt/cephfs/fio, and specify a filename in the job file generated by pbench-fio. There is no --filename parameter. If you try to specify it with -d then the --directory parameter does not get used.

This does not impact Cinder volume testing of OpenStack.

Here's an example of what does work for distributed filesystem testing using fio:

-=>>cat f2.job
[global]
bs=1024k
ioengine=sync
ramp_time=5

[job-/mnt/cephfs/fio/ben]
filename=ben
directory=/mnt/cephfs/fio
rw=write
size=104857600

# fio --client=~/osds.list
...
# ls -l /mnt/cephfs/fio
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.229.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.232.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.235.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.240.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.242.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.245.ben
-rw-r--r--. 1 root root 104857600 Sep 14 17:23 10.12.64.248.ben

Note that fio --client can use separate files for each host. The PR for this was fio PR 66 and it is described in preceding PR 65.

If I try to run this in pbench-fio with something like:

pbench-fio -t write --directory=/mnt/cephfs/fio -d ben --client-file=/root/osds.list 

We don't even get files in the Cephfs filesystem, they go to /tmp/fio or some default place instead. However,

pbench-fio -t write -d /mnt/cephfs/fio/ben --client-file=/root/osds.list

Will produce just one file for the entire distributed filesystem and all clients will access the same file. While this is an interesting test, it is not typically the one we want to do.

[atheurer]

So you want 1) a new option in pbench-fio for --file= 2) when the option is used, pbench-fio should populate the fio job file with "filename=

....and fio --client will take care of adding a unique string to the file name specified in the job file, so the client don;t use the same file?

[portante]

@bengland2, can you update where you are at with this?

[bengland2]

sorry I've been a poor correspondent. But yes, the original idea of fio --client enhancement was to generate unique filenames across a cluster, so if we specify a filename=ben we should see files like ben.172.18.0.1, ben.172.18.0.2, ... etc. I tried this on a ceph cluster with fio --client back in June. So fio --client support is there, but the parameters in pbench-fio are not AFAIK.

[bengland2]

@ndokos I just raised the priority on this issue. This is why we can't use pbench-fio for cephfs testing or glusterfs testing, or OpenStack Manila testing.