aurproxy is a load balancer manager with knowledge of Apache Aurora's service discovery mechanism and integration with Aurora's task lifecycle. It is a python application that manages a fully featured but not Aurora-aware load balancer (currently only nginx is supported). When Aurora service discovery events occur, aurproxy detects them, rewrites the load balancer's configuration file and then triggers a graceful restart of the load balancer. Use aurproxy to expose dynamic Aurora service endpoints to services and applications that don't have knowledge of Aurora's service discovery mechanism.
Replace your zk_servers, role, environment, job, etc. in the value of the "config" argument below:
# 1 - Build and push container to your registry - replace registry address.
PACKAGE_VERSION=20150430.0; docker build -t docker.mydomain.com/library/aurproxy:$PACKAGE_VERSION . && docker push docker.mydomain.com/library/aurproxy:$PACKAGE_VERSION
# 2 - Launch container
docker run -t -i --net=host docker.mydomain.com/aurproxy:20150430.0 bash
# 3 - Run setup to configure load balancer
cd /opt/aurproxy && \
python -m "tellapart.aurproxy.command" run \
--setup \
--management-port=31325 \
--config '{"backend": "nginx", "servers": [{"routes": [{"sources": [{"endpoint": "http", "zk_servers": "0.zk.mycluster.mydomain.com:2181,1.zk.mycluster.mydomain.com:2181,2.zk.mycluster.mydomain.com:2181", "environment": "devel", "job": "web", "role": "myrole", "source_class": "tellapart.aurproxy.source.AuroraProxySource"}], "locations": ["/"]}], "hosts": ["default"], "ports": [8080], "context": {"default_server": "True", "location_blacklist": ["/health", "/quitquitquit", "/abortabortabort"]}}]}'
# 4 - Start load balancer
/usr/sbin/nginx -c /etc/nginx/nginx.conf &
# 5 - Re-run #2 without --setup flag.
# 6 - Test from another shell - assuming host networking
# (otherwise you'll have to map port through in #1 above)
# and something to point to in configured ProxySource.
curl 127.0.0.1:8080/robots.txtSet up Aurora devcluster
Build and push container to your registry - replace registry address.
PACKAGE_VERSION=20150430.0; docker build -t docker.mydomain.com/library/aurproxy:$PACKAGE_VERSION . && docker push docker.mydomain.com/library/aurproxy:$PACKAGE_VERSIONEdit example/aurproxy_hello_world.aur to match the information for the docker image above.
Copy example/aurproxy_hello_world.aur, example/hello_world.aur, example/hello_world.py to your local aurora source directory (will show up in "/vagrant/" in devcluster).
"vagrant ssh" into aurora devcluster instance.
Install hello_world.py prerequisites:
apt-get install python-pip
pip install Flask==0.10.1Create jobs:
aurora job create devcluster/www-data/devel/hello_world /vagrant/hello_world.aur
aurora job create devcluster/www-data/devel/aurproxy /vagrant/aurproxy_hello_world.aurWait for the jobs to come up - the first download of the aurproxy docker image may take a while:
http://192.168.33.7:8081/scheduler/www-data/devel/aurproxyTest from host:
curl 192.168.33.7:8080
# Expect 200 "Hello World!"Find running aurproxy task instance in Aurora web interface, open stderr of "aurproxy" process.
Restart hello_world job while watching aurproxy stderr log to see proxy configuration update in action:
aurora job restart devcluster/www-data/devel/hello_worldAurproxy HTTP traffic mirroring and replay features use gor, which can be set up to mirror a fixed number of queries per second from a full aurproxy task instance over a TCP stream to one or more gor replay servers. Aurproxy finds gor instances using sources, and will update its gor command line to add new endpoints as they appear and remove old ones as they disappear.
To use traffic mirroring:
Set up a gor replay server. It can be but doesn't have to be in Aurora. See "traffic replay" for instructions on how to set one up using aurproxy to manage it.
Add a gor_process to your Aurproxy task definition.
# max_failures=0 is important:
# aurproxy kills the gor process when it updates mirror.sh,
# and max_failures=0 means that Aurora won't treat the task as
# unhealthy after some number of gor process restarts.
gor_process = Process(
name='gor',
cmdline='/etc/aurproxy/gor/dynamic.sh',
max_failures=0)Pass values for mirror_source (an aurproxy source configured to point to your gor replication server(s)), mirror_ports, mirror_max_qps, and mirror_max_update_frequency into aurproxy when starting it.
To use traffic replay:
Set up a replay job to run aurproxy "run_replay" with the "--setup" flag and then both run_replay and gor, as per above in the traffic mirroring instructions. run_replay command line example:
cd /opt/aurproxy && \
python -m tellapart.aurproxy.command run_replay \
--management-port 12345 \
--replay-port 12346 \
--replay-source '{"name": "replay", "source_class": "tellapart.aurproxy.source.ApiSource"}' \
--replay-max-qps 1000Build the image. EG:
docker build -t docker.mydomain.com/aurproxy:latest .pip install virtualenv
virtualenv -p python2.7 --no-site-packages --distribute ~/.virtualenvs/aurproxy
cd ~/src/aurproxy
source ~/.virtualenvs/aurproxy/bin/activate
pip install -r ~/src/aurproxy/requirements.txt
pip install -r ~/src/aurproxy/requirements_dev.txtnosetests --all-modules --where ./tellapart --with-coverage --cover-package=tellapart.aurproxy --cover-inclusiveaurproxy is designed to support using different load balancers as backends. Within aurproxy, a backend extends aurproxy.backends.ProxyBackend and is where the loadbalancer-specific logic and management code should live. nginx is currently the only backend implementation, but an HAProxy backend implementation should be possible.
Endpoints are host+port pairs that represent a running task instance. They don't have to be hosted in Aurora.
Sources are classes that are responsible for providing, maintaining, and signaling updates to a list of endpoints.
Active Registration refers to registration events that take place during the normal Aurora lifecycle for Aurproxy task instances - on startup, registration is triggered (if configured), and on graceful shutdown, deregistration is triggered (if configured).
ELB
--registration-arg=region=us-east-1
--registration-arg=elb_names=MyElb
--registration-class=tellapart.aurproxy.register.elb.ElbSelfRegistererRoute53
--registration-arg=region=us-east-1
--registration-arg=domain=myproxy.mydomain.com
--registration-arg=hosted_zone_id={{hosted_zone}}
--registration-arg=ttl=60
--registration-class=tellapart.aurproxy.register.route53.Route53SelfRegistererSynchronization refers to a way to run a registration plugin as an administrative script or as an Aurora cron job. It is intended to both remove outdated aurproxy tasks left behind by slaves that may have disappeared or on which task shutdown was not graceful as well as add missing any aurproxy tasks o supported systems (EG: other load balancers or DNS).
ELB
cd /opt/aurproxy && python -m "aurproxy.command" synchronize \
--registration-arg=region=us-east-1 \
--registration-arg=elb_names=MyElb \
--registration-class=tellapart.aurproxy.register.elb.ElbJobRegistererRoute53
cd /opt/aurproxy && python -m "aurproxy.command" synchronize \
--registration-arg=region=us-east-1 \
--registration-arg=domain=myproxy.mydomain.com \
--registration-arg=hosted_zone_id={{hosted_zone}} \
--registration-arg=ttl=60 \
--registration-class=tellapart.aurproxy.register.route53.Route53JobRegistererAurproxy supports recording:
Librato
--metric-publisher-class=tellapart.aurproxy.metrics.publisher.LibratoMetricPublisher \
--metric-publisher-arg=source={{cluster}}.{{role}}.{{environment}}.{{job}} \
--metric-publisher-arg=period=60 \
--metric-publisher-arg=api_user={{librato_user}} \
--metric-publisher-arg=api_token={{librato_token}}OpenTSDB
--metric-publisher-class=tellapart.aurproxy.metrics.publisher.OpenTSDBMetricPublisher \
--metric-publisher-arg=source={{cluster}}.{{role}}.{{environment}}.{{job}} \
--metric-publisher-arg=period=60 \
--metric-publisher-arg=prefix=apps. \
--metric-publisher-arg=host=localhost \
--metric-publisher-arg=port=4242Currently, the only log aggregation implementation is Sentry. If you pass a sentry_dsn in via the command line, logged exceptions will be recorded to Sentry.
--sentry-dsn='gevent+https://{{sentry_user}}:{{sentry_pass}}@app.getsentry.com/{{sentry_project}}'Some of the configuration elements above support a "context" dictionary. This a "bucket of stuff" that gets passed directly down to the configuration rendering context and is the place to put backend-specific configuration values (EG: "default_server": True)
import json
cmd = 'cd /opt/aurproxy && python -m "tellapart.aurproxy.command" run ' \
'--setup --management-port=31325 --max-update-frequency=10 ' \
'--update-period=2 --sentry-dsn='' --config \''
config = {
"servers": [
{
"routes": [
{
"sources": [
{
"endpoint": "http",
"announcer_serverset_path": "sd/mycluster",
"job": "myjob",
"environment": "prod",
"zk_servers": "0.zk.mycluster.mydomain.com:2181,"
"1.zk.mycluster.mydomain.com:2181,"
"2.zk.mycluster.mydomain.com:2181",
"role": "myrole",
"source_class": "tellapart.aurproxy.source.AuroraProxySource",
"share_adjusters": [
{
"share_adjuster_class": "tellapart.aurproxy.share.adjusters.RampingShareAdjuster",
"ramp_seconds": 60,
"ramp_delay": 10,
"update_frequency": 10,
"curve": "linear"
},
{
"share_adjuster_class": "tellapart.aurproxy.share.adjusters.HttpHealthCheckShareAdjuster",
"route": "/health",
"interval": 3,
"timeout": 2,
"unhealthy_threshold": 2,
"healthy_threshold": 2,
}
]
}
],
"locations": [
"/"
]
}
],
"healthcheck_route": "/healthcheck",
"hosts": [
"default"
],
"ports": [
10080
],
"context": {
"default_server": "True",
"location_blacklist": ["/health", "/quitquitquit", "/abortabortabort"]
}
}
],
"backend": "nginx"
}
cmd += json.dumps(config) + '\''
print cmd