Benchmark NFS vs rsync on inotify, GlusterFS, etc.

[geerlingguy]

NFS seems to add a little network overhead, though I don't think it's as much as I'm expecting. Might be interesting—especially since this site isn't heavy on writes/reads—to see if just having an inotify-based rsync configuration or even getting extravagant and using GlusterFS would speed up a shared files directory over the existing NFS implementation.

Plus it might simplify the config in general; right now I have an NFS mount shared from the cache/Redis server, and NFS has to connect from each of the web heads.

See:

• inotify and rsync on large number of files
• inosync
• GlusterFS (and my burgeoning Galaxy role — and the new gluster_volume module!)

[geerlingguy]

I have GlusterFS working alongside NFS, so I should be able to start moving files back and forth and benchmarking. A few useful commands (though I'm automating everything in playbooks, so this shouldn't be needed except when debugging):

# Manually mount 'gluster' volume from any Gluster server to /mnt/gluster.
sudo mount -t glusterfs 10.0.1.61:/gluster /mnt/gluster

# Unmount the mounted volume (need the -l 'lazy' option).
sudo umount -l /mnt/gluster

# Stop and delete the 'gluster' volume.
sudo gluster volume stop gluster
sudo gluster volume delete gluster

# Switch to other mount (assuming files have been synced between them).
sudo rm -rf files && sudo ln -s /mnt/gluster/files/ files && ls -lah

[geerlingguy]

NFS mounted directory:

$ ab -n 100 -c 10 -C "SESS..." http://pidramble.com/about
This is ApacheBench, Version 2.3 <$Revision: 1554214 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking pidramble.com (be patient).....done

Server Software:        nginx/1.2.1
Server Hostname:        pidramble.com
Server Port:            80

Document Path:          /about
Document Length:        5775 bytes

Concurrency Level:      10
Time taken for tests:   10.645 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      694000 bytes
HTML transferred:       577500 bytes
Requests per second:    9.39 [#/sec] (mean)
Time per request:       1064.529 [ms] (mean)
Time per request:       106.453 [ms] (mean, across all concurrent requests)
Transfer rate:          63.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.2      1       2
Processing:   845 1021  63.8   1011    1239
Waiting:      844 1020  63.8   1011    1238
Total:        846 1021  63.8   1012    1241

Percentage of the requests served within a certain time (ms)
  50%   1012
  66%   1041
  75%   1056
  80%   1066
  90%   1100
  95%   1148
  98%   1209
  99%   1241
 100%   1241 (longest request)

GlusterFS mounted directory:

$ ab -n 100 -c 10 -C "SESS..." http://pidramble.com/about
This is ApacheBench, Version 2.3 <$Revision: 1554214 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking pidramble.com (be patient).....done

Server Software:        nginx/1.2.1
Server Hostname:        pidramble.com
Server Port:            80

Document Path:          /about
Document Length:        5775 bytes

Concurrency Level:      10
Time taken for tests:   11.783 seconds
Complete requests:      100
Failed requests:        0
Total transferred:      694000 bytes
HTML transferred:       577500 bytes
Requests per second:    8.49 [#/sec] (mean)
Time per request:       1178.287 [ms] (mean)
Time per request:       117.829 [ms] (mean, across all concurrent requests)
Transfer rate:          57.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.2      1       2
Processing:   932 1161  69.9   1165    1449
Waiting:      932 1161  70.0   1165    1449
Total:        933 1162  70.0   1166    1451

Percentage of the requests served within a certain time (ms)
  50%   1166
  66%   1189
  75%   1203
  80%   1218
  90%   1242
  95%   1258
  98%   1295
  99%   1451
 100%   1451 (longest request)

I ran both tests at least 3 times, and the results I care about most are below:

Type	Req/s
GlusterFS	8.49
NFS	9.39

The GlusterFS mount resulted in extremely consistent results (all three runs were within 0.50 req/sec of each other). Additionally, when switching back to the NFS mount from Gluster, it seems something in the symlinking or somewhere must've caused some delay, because later tests were only getting ~6.5 req/sec through NFS. I'm guessing a reboot would get things back into order.

[geerlingguy]

Because using GlusterFS gives us triple redundancy, and seems to be a lot more modern approach to filesystem clustering/distribution on the webservers—and because it allows us to not have to use another one of the servers as an NFS server—I'm going to stick with GlusterFS for now. Creating a new issue to capture the commits.