search

weaveworks-experiments / flux-classic

Microservice routing
http://weaveworks.github.io/flux/
Apache License 2.0
82 stars 10 forks source link

Seamless edge load-balancing into Weave Net (in AWS/ECS) #31

Open 2opremio opened 8 years ago

2opremio commented 8 years ago

TL;DR

Users of Weave Net often ask us how to expose and load-balance requests to containers from the outside world. Here are a few of those cases:

https://groups.google.com/a/weave.works/forum/#!topic/weave-users/TlaXW-KNqbw https://groups.google.com/a/weave.works/forum/#!searchin/weave-users/expose/weave-users/n0QB7Nmp738/GyLAqQqUBgAJ https://github.com/weaveworks/weave/issues/1211

I found myself thinking about this situation today. This issue is going to be a write-up of my best bet to solve it (without Flux), how I think it could be solved today with Flux (which so far I have discarded) and how I would have liked Flux to solve it in order to make it my preferred solution.

User Story

I am going to illustrate this issue with a specific (and real) example.

As a user, I have a PHP website, and I want to load-balance external requests into stateless PHP processes. The PHP processes have been containerized and are being transitioned to run as task instances of an Amazon ECS service. The tasks are running in a Weave Network. The main reasons for choosing Weave Net have been:

  1. Easy communication with other backend components, some of which will still live on-premise.
  2. Flexible autoscaling. Weave allows deploying multiple PHP containers in the same VM, all listening on TCP port 80, without worrying about port clashes (i.e. no need to stick to 1 container per VM or port-mapping schemes).

The last piece of the puzzle is load-balancing external requests into the PHP containers. The obvious tool to do this in AWS is an ELB, which I also want to use to trigger autoscaling events for expanding/reducing my cluster. However, it's not obvious how to load-balance the requests into the weave network and, in particular, there is no direct integration with ELBs.

My best bet to solve it (without Flux)

Weave Net suggests combining weave expose+ iptables as a means to access containers externally but it's clearly insufficient (it doesn't help with load-balancing and it doesn't provide a way to automatically book-keep the rules when new PHP-containers are added/are removed/die).

I decided to do the following:

  1. Enable WeaveDNS client-based load-balancing of PHP-containers: I assigned the same phpfrontend.weave.local record to all my PHP containers.

  2. To interface into the Weave Network from the outside world, in every VM, I deployed multiple (for redundancy) ngninx containers with the following configuration:

    server {
listen 80;
resolver 172.17.42.1:53;
location /api/users/ {
   proxy_pass http://phpfrontend.weave.local$request_uri;
}
}

    This nginx container is part of the Weave network and is going to listen on the hosts port 80 (i.e. Docker's -p 80:80).

    To load-balance requests into each machine, the nginx containers are deployed as an ECS service with an ELB (see http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html and http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html). As a result, each nginx container will get automatically attached to the ELB when deployed receiving external requests.

    How I would had solved it with Flux today (and why I discarded it)

I had never used Flux, but it conceptually feels like it should be able to provide a simple solution, well integrated with Weave Net.

However, I only came up with something surprisingly similar to the solution above but with extra moving pieces (Flux agents + etcd), one extra step (service creation) and no obvious advantage :

  1. Create a service: phpfrontend

    $ fluxctl service phpfrontend --address 10.128.0.1:80 --protocol=http
$ fluxctl select hello default --image acmeorg/phpfrontend --port-fixed 80

  2. Similarly to the solution above, create an ECS service with an ELB deploying an multiple nginx edge balancers. For clarity, the manual equivalent of this would be running

    docker run --privileged --net=host -d -e ETCD_ADDRESS -e SERVICE=phpfrontend weaveworks/flux-edgebal

    on each host (and attaching the ELB).

    How I would had liked Flux to solve it

In order to provide a better experience the process needs to be simpler and better integrated with Weave Net:

  1. Inferring services from WeaveDNS records (something like fluxctl select phpfrontend default --weavedns phpfrontend.weave.local) is a natural extension to how services have been internally load-balanced in Weave before Flux.
  2. Provide an embedded edge balancer. Having to manually manage the edge load balancer is inconvenient. A command like fluxctl expose phpfrontend -p 80 which exposes service phpfrontend through port 80 in every host would be very helpful in this case.
    • The embedded edge load-balancer doesn't need to provide all the features of nginx and in fact the user can revert to external load balancer like weaveworks/flux-edgebal when needed.
    • The --weavedns rules could optionally be efficiently implemented using iptables, as explained in Weave Net's Service Export .

Also, directly exposing services through ELBs would be very valuable to AWS users. See https://groups.google.com/a/weave.works/forum/#!searchin/weave-users/nginx/weave-users/4IfS9SDDYEw/KLywaBBUAwAJ

The expose command could be extended to automatically create an ELB (or equivalent for other cloud providers), and directly attach the service to the ELB without intermediate edge load-balancers: e.g. fluxctl expose phpfrontend --external. This is somewhat the equivalent of type: LoadBalancer in Kubernetes and, in the same way, Flux would need to be aware of the cloud provider it's being executed in.

As an extra benefit, in the case of AWS, fluxctl expose <service> --external lifts the dependency on ECS services/Kubernetes services to map ELBs to containers, which may be appealing to some users.

Finally, as an optimization, it would be great if both Flux and the edge-load balancer (embedded/ELB-based or else) could provide locality-aware load-balancing to avoid cross-traffic between instances. For instance, something like fluxctl service phpfrontend --local --address 10.128.0.1:80 --protocol=http

Putting everything together

With all the proposed features, adding an edge balancer into the PHP-containers in the Weave network would be as simple as:

$ fluxctl service phpfrontend --local --address 10.128.0.1:80 --protocol=http
$ fluxctl select phpfrontend default --weavedns phpfrontend.weave.local
$ fluxctl expose phpfrontend --external
rade commented 8 years ago

The reason for deploying it in every VM is that as side-effect of getaddrinfo() (weaveworks/weave#1245), ngnix should favor PHP-containers living in the same machine, making it locality-aware and avoiding cross traffic.

That's not how it works. The getaddrinfo() logic picks the IP with the longest common prefix to the local IP. In case of weave net, that has nothing whatsoever to do with locality.

2opremio commented 8 years ago

That's not how it works. The getaddrinfo() logic picks the IP with the longest common prefix to the local IP. In case of weave net, that has nothing whatsoever to do with locality.

You are right, I will edit the issue to reflect this. However, I believe locality-awareness would still be a desired feature for load-balancing.

2opremio commented 8 years ago

Related: http://kubernetes.io/docs/user-guide/ingress (k8s equivalent of weaveworks/flux-edgebal)