Performance Upgrades - Githubissues

state-hiu / ittc-server-django

Front-End ITTC Server with Proxy, Cache, Logs, and Stats

MIT License

0 stars 3 forks source link

Performance Upgrades #13

Open state-hiu opened 9 years ago

state-hiu commented 9 years ago

django-memcached-pool

https://github.com/mozilla/django-memcached-pool https://github.com/state-hiu/django-memcached-pool/tree/ittc https://github.com/mozilla/django-memcached-pool/pull/14

[x] Pool in unmerged PRs and create new fork

Gunicorn

[x] Optimize worker_connections / --worker-connections. Default is 1000, try 2000 to start.
[x] max_requests at 10,000 to mitigate memory leaks / http://gunicorn-docs.readthedocs.org/en/latest/settings.html

Gevent

[x] Use Gevent monkey patching for Mongo Clients (http://api.mongodb.org/python/current/examples/gevent.html)
[x] Use Gevent monkey patching for tile requests (https://github.com/gevent/gevent/blob/master/examples/concurrent_download.py)
[ ] Use most effective Gevent patching (see http://gevent.org/whatsnew_1_0.html and https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/ggevent.py#L57). patch_all(subprocess=True) and https://github.com/mongodb/mongo-python-driver#testing

Celery

[ ] Use celery to write back direct tile to cache and return tile immediately

Visibility

[x] Show cache size in info page. From command line that is: (echo stats | nc 127.0.0.1 11212 | grep "STAT bytes ")
[ ] Show number of connections / max pool size

Queue Expiration http://www.imankulov.name/posts/celery-messages-ttl.html

[x] Set expiration time for indirect tile requests to 60 seconds. Therefore, after spike in requests, queue will return to normal.

Images

[ ] Strip alpha band out of pngs if not needed. Create separate function and do right after tilesource.requestTile. Something like tile = stripAlpha(tile). Comment: actually doesn't seem to decrease size substantially.

Logging

[x] Set log level for Gunicorn to critical (--log-level=critical)

Memcached

[ ] Increase number of threads. Default is only 4. Try 8 then 16 for memcached-tiles. See https://docs.oracle.com/cd/E17952_01/refman-5.1-en/ha-memcached-using.html https://github.com/memcached/memcached/blob/master/doc/protocol.txt. Decrease threads for default and celery results caches.

RabbitMQ

[x] ulimit -n 1024 in /etc/default/rabbitmq-server