HCI Stress Testing

[kousu]

We are unsure about performance of our storage system. We want to measure I/O performance to duke.neuro.polymtl.ca.

There are several factors:

• which disk we're writing to (represented by different mountpoints)
• whether we're writing from
  • locally on duke
  • remotely on the neuropoly LAN
    • over smb://
      • from Windows
      • from linux's mount -t cifs
    • over afp://
  • remotely over the VPN
    • over smb://
      • from Windows
      • from linux's mount -t cifs
    • over afp://
• the I/O pattern:
  • large bulk writes
  • large bulk reads
  • many small writes
  • many small reads
  • interleaved bulk/small writes

[kousu]

There are several tooling recommendations on https://wiki.archlinux.org/index.php/Benchmarking

• https://github.com/Algodev-github/S
• http://www.phoronix-test-suite.com/
• https://en.wikipedia.org/wiki/Bonnie%2B%2B
• simply using dd + iostat

https://www.thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_dd is a guide to testing with dd.

https://linuxreviews.org/HOWTO_Test_Disk_I/O_Performance claims dd is misleading and you should use fio instead.

[kousu]

Anything we come up with here could be applied to data.neuro.polymtl.ca too

[alexfoias]

@kousu do we have an initial plan for testing the HCI on duke?

[kousu]

No. The plan is to try running some of those tools above. That's as far as I've gotten.

[alexfoias]

Test over VPN, mount made with AFP from OSX 11.2.1 (20D74):

HCI:

dd if=/dev/zero of=/Volumes/histology/test_HCI/tempfile bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 215.865430 secs (4974126 bytes/sec)

Old duke (/temp partition):

dd if=/dev/zero of=/Volumes/temp/test_HCI/tempfile bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 216.610216 secs (4957023 bytes/sec)

[alexfoias]

From joplin:

HCI:

(base) alfoi@joplin:~/duke/histology/test_HCI$ dd if=/dev/zero of=/home/GRAMES.POLYMTL.CA/alfoi/duke/histology/test_HCI/tempfile bs=1M count=1024 conv=fdatasync,notrunc status=progress
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 21.532 s, 49.9 MB/s

Old duke (/temp partition):

(base) alfoi@joplin:~/duke/histology/test_HCI$ dd if=/dev/zero of=/home/GRAMES.POLYMTL.CA/alfoi/duke/temp/test_HCI/tempfile bs=1M count=1024 conv=fdatasync,notrunc status=progress
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.3426 s, 94.7 MB/s

[alexfoias]

Test with smaller files.

[alexfoias]

@kousu @taowa could you please try to see if you have WR permissions on duke/histology in order do more testing on the volume? you can try to write in /duke/histology/test_HCI

[taowa]

Just checked- I have access to it through my membership in neuropoly_admin.

Taowa

On Jul 30, 2021, at 11:29, Alexandru Foias @.***> wrote:

@kousu @taowa could you please try to see if you have WR permissions on duke/histology in order do more testing on the volume?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[alexfoias]

Great news. @kousu Listed above some options. I think we need to put in place a script/procedure to properly compare the WR speed on duke.

We need to test the WR difference between: duke/temp (this share point is on a physical volume on duke) duke/histology (this share point is on a HCI volume that is mounted on duke)

[alexfoias]

@kousu I feel like we can close this issue.