geerlingguy / raspberry-pi-dramble

DEPRECATED - Raspberry Pi Kubernetes cluster that runs HA/HP Drupal 8
http://www.pidramble.com/
MIT License
1.67k stars 261 forks source link

Benchmark various network configurations #8

Closed geerlingguy closed 9 years ago

geerlingguy commented 9 years ago

tl;dr Getting Gigabit Networking on a Raspberry Pi 2 and B+


I'd like to see what gives us the most bang for the buck, especially with regard to the load balancer and database server, which will need the lowest latency and highest throughput.

Theories/questions to test/verify/debunk

Switching interfaces (via SSH, since I'm doing everything headless) is a simple matter of:

  1. Log into the interface you want to test (e.g. IP .35 for Wifi, .36 for Ethernet, in my case).
  2. $ sudo ifdown [eth0|wlan0] (whichever one you are not connected through).
  3. $ sudo ifconfig -a to list all interfaces and verify current status.
  4. $ ping 8.8.8.8 to test Internet connectivity, and if it's down:
    1. $ ip route show to make sure there's a default route configured.
    2. If not, $ sudo ip route add default via 10.0.1.1 dev [eth0|wlan0] (whichever one you are connected through).
    3. $ ping 8.8.8.8 again to test Internet connectivity.
  5. Profit!

    Benchmarks

    Internal microSD vs external USB 3.0 SSD

    10/100 Ethernet (built-in)

Using /tmp (on the internal Kingston Class 10 microSD card):

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-15 15:04:17 (6.05 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.36:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  2954570.28 bytes/sec (2.94MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.36:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  3555499.69 bytes/sec (3.55MB/sec)

Using /ssd (on the external USB 3.0 SSD):

$ wget -O /ssd/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-15 15:18:22 (6.32 MB/s) - `/ssd/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.36:/ssd/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  2796993.20 bytes/sec (2.80MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.36:/ssd/test100.zip
sent 104887199 bytes  received 42 bytes  3555499.69 bytes/sec (3.55MB/sec)

By testing on an SSD and on the microSD card, it seems pretty obvious that disk I/O is not the bottleneck here, but rather the entire bus (so it seems).

And it also seems likely that, with a decent enough microSD card, there's no real performance to be gained (at least network/throughput-wise) from using an external HDD or SSD. I'm going to also test random read/write scenarios over in #7, so that might shed more light on database/codebase activity, and optimizations to be had there...

Note also that my local Internet connection is a bit past saturating the Pi's network interface (heck, it requires Gigabit or 802.11ac with a strong signal to saturate the connection on my Macs!). Here's the initial download, running on my MacBook Air with 802.11ac a couple feet from my AirPort Extreme:

$ wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-15 16:01:04 (8.77 MB/s) - '/dev/null' saved [104874307/104874307]

Network throughput in common usage scenarios

802.11n USB WiFi

Using /tmp (on the internal Kingston Class 10 microSD card):

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-15 15:32:44 (2.94 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.35:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  2589808.52 bytes/sec (2.59MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.35:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  2873623.04 bytes/sec (2.87MB/sec)

WiFi is slightly slower; the signal was great, and I think it was hitting the full bus speed here as well (just like with the 10/100 example). But pretty stable and fast regardless. I have no hesitation using a reliable little WiFi adapter in lieu of wired Ethernet when it's more convenient.

10/100 Ethernet (built-in)

Using /tmp (on the internal Kingston Class 10 microSD card):

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-15 15:59:04 (2.69 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.37:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  2655373.29 bytes/sec (2.66MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.37:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  3438925.93 bytes/sec (3.43MB/sec)

10/100/1000 Gigabit Ethernet (external USB dongle)

Using /tmp (on the internal Kingston Class 10 microSD card):

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-16 21:07:19 (6.32 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.37:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  2954570.28 bytes/sec (2.95MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.37:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  3438925.93 bytes/sec (3.44MB/sec)

Raw Network Throughput using iperf

To do this test, install iperf on the Pi ($ sudo apt-get install -y iperf), then:

  1. (On the Pi) Start iperf: $ iperf -s
  2. (On other machine) Start the test: $ iperf -c 10.0.1.37

    10/100 Ethernet

$ iperf -c 10.0.1.36
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   113 MBytes  94.4 Mbits/sec

802.11n USB WiFi

$ iperf -c 10.0.1.35
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  53.1 MBytes  44.5 Mbits/sec

10/100/1000 Gigabit Ethernet (external USB dongle)

$ iperf -c 10.0.1.38
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   266 MBytes   222 Mbits/sec

Conclusion

It looks like, at least as far as a Raspberry Pi B+ and A+ are concerned, internal 10/100 Ethernet is more than adequate for most use cases, and other interfaces add throughput, but only for operations where other I/O is not a priority.

If network throughput is a priority, it is definitely worth investing in a 10/100/1000 Gigabit USB 3.0 adapter; even over the Raspberry Pi's USB 2.0 bus, you will see at least double the bandwidth, meaning uploads, downloads and streaming will get a LOT more bandwidth!

geerlingguy commented 9 years ago

I've posted a summary of my findings here: Getting Gigabit Networking on a Raspberry Pi

geerlingguy commented 9 years ago

Raspberry Pi 2 results

10/100 Ethernet (built-in)

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-18 21:53:58 (8.54 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.39:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  7769425.56 bytes/sec (7.77MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.39:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  9120629.65 bytes/sec (9.12MB/sec)
$ iperf -c 10.0.1.38
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   113 MBytes  94.8 Mbits/sec

The onboard LAN integration has dramatically improved in the model 2! I was blown away by the difference, thinking I must've used a different SD card, or something, in the B+ testing. But I then ran all the tests twice more on the B+ and Pi 2, and found that they're correct. WOW.

My best guess is that the onboard LAN is no longer shared with the same bus/data pipeline as the microSD card, meaning data can be read/written to the microSD card independently of the LAN connection bandwidth.

Gigabit USB Ethernet adapter

$ wget -O /tmp/test100.zip http://speedtest.wdc01.softlayer.com/downloads/test100.zip
2015-02-18 21:43:08 (6.11 MB/s) - `/tmp/test100.zip' saved [104874307/104874307]

$ rsync --progress pi@10.0.1.38:/tmp/test100.zip ~/Downloads/test100.zip
sent 42 bytes  received 104887203 bytes  8390979.60 bytes/sec (8.39MB/sec)

$ rsync --progress ~/Downloads/test100.zip pi@10.0.1.38:/tmp/test100.zip
sent 104887199 bytes  received 42 bytes  6356802.48 bytes/sec (6.36MB/sec)
$ iperf -c 10.0.1.38
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   255 MBytes   214 Mbits/sec
geerlingguy commented 9 years ago

Moved further discussion to the wiki: https://github.com/geerlingguy/raspberry-pi-dramble/wiki/Networking-Benchmarks