performance testing and improvement thoughts

This is mostly to investigate if uvloop is actually as awesome as claimed.

Python 3.7 on OSX 10.13, Macbook Pro 2015 with i7 4-core/8-thread at 2.2 Ghz and 16 GB

pip freeze:

astrobase==0.3.16
astropy==3.0.3
bleach==2.1.3
certifi==2018.4.16
chardet==3.0.4
cycler==0.10.0
html5lib==1.0.1
idna==2.7
itsdangerous==0.24
jplephem==2.8
kiwisolver==1.0.1
lcc-server@d603197b94111a1c8e16167381017eac1e8c9e0d
Markdown==2.6.11
matplotlib==2.2.2
numpy==1.15.0
passlib==1.7.1
Pillow==5.2.0
psutil==5.4.6
pyeebls==0.1.6
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.7.3
pytz==2018.5
requests==2.19.1
scikit-learn==0.19.2
scipy==1.1.0
six==1.11.0
tornado==5.1
tqdm==4.23.4
urllib3==1.23
uvloop==0.11.0
webencodings==0.5.1

Running indexserver with the default 4 background workers, with logging set to a file instead of stdout.

Before uvloop:

@nerrivik:~/scratch
[23:49]$ wrk -t12 -c400 -d30s http://127.0.0.1:12500/api/datasets
Running 30s test @ http://127.0.0.1:12500/api/datasets
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   775.47ms  154.82ms 962.30ms   86.16%
    Req/Sec    44.35     38.09   240.00     81.64%
  14145 requests in 30.10s, 370.63MB read
  Socket errors: connect 0, read 441, write 0, timeout 0
Requests/sec:    469.89
Transfer/sec:     12.31MB

After uvloop:

[23:50]$ wrk -t12 -c400 -d30s http://127.0.0.1:12500/api/datasets
Running 30s test @ http://127.0.0.1:12500/api/datasets
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   713.53ms  145.67ms 989.02ms   85.81%
    Req/Sec    52.22     45.63   250.00     77.57%
  15316 requests in 30.08s, 401.31MB read
  Socket errors: connect 0, read 464, write 0, timeout 0
Requests/sec:    509.11
Transfer/sec:     13.34MB

Slight improvement I guess. Will continue to investigate.

waqasbhatti / lcc-server

performance testing and improvement thoughts #17