Setting NumberOfPhysicalBreaks = 0x21 is making zvol perform better when the blockssize is >=128K

[arun-kv]

By changing 'pConfigInfo->NumberOfPhysicalBreaks = 0x21' we are able to improve ZFSin performance by 50%(for >=128K workloads). Default 'NumberOfPhysicalBreaks' is set to 0x11, that is 64K, so for every 128K read/write storport will send two request to ZFSin, now changing the value to 0x21 reduced the number of request by half and thus improving the performance. Also we can see that the performance is now exactly matching with the Linux performance.

Below is the vdbench output for 100% 128K write workload:

Windows:
---------------------------------------------------------------------------------------------------------------------------------------
Disk Type               Operation       i/o       MB/sec    queue  cpu%    cpu%    
                                            rate      1024**2   depth  sys+u   sys     
---------------------------------------------------------------------------------------------------------------------------------------
Windows:
zvol(with 1vdev)            Write(100%)      1152.4   144.06    15.9   8.1   7.8
zvol(with 2vdev)            Write(100%)      2162.3   270.29    15.9  13.1  12.6
zvol(with 3vdev)            Write(100%)      2996.9   374.62    15.9  15.5  15.1    

zvol(with 1vdev)            Write(100%)      3107.4   **388.42**    15.8   9.9   9.2    NumberOfPhysicalBreaks = 0x21
zvol(with 2vdev)            Write(100%)      6148.6   **768.57**    15.7  14.1  13.2    NumberOfPhysicalBreaks = 0x21
zvol(with 3vdev)            Write(100%)      7930.1   **991.26**    13.7  13.7  12.9    NumberOfPhysicalBreaks = 0x21

Linux:

zvol(with 1vdev)            Write(100%)      3046.2   380.77    15.9   6.3   5.8
zvol(with 2vdev)            Write(100%)      6134.4   766.80    15.7  12.2  11.2
zvol(with 3vdev)            Write(100%)      7882.3   985.28    14.0  17.3  16.0

[lundman]

Oh that is awesome, don't think I would ever have found that. Thanks for sharing!