Support load balancing with nginx dynamic upstreams

[subnetmarco]

Support for dynamic upstreams that will enable dynamic load balancing per API.

 # sample upstream block:
 upstream backend {
    server 127.0.0.1:12354;
    server 127.0.0.1:12355;
    server 127.0.0.1:12356 backup;
}

So we can proxy_pass like:

proxy_pass http://backend;

[bobrik]

I'd love to see that too. My use-case is routing requests to dynamic mesos tasks with zoidberg, kong would be a good candidate to do the routing part. I was going to use nginx with 2 ports anyway. Let me know if it makes sense to use kong for this.

Here are the options I see:

1. openresty/lua-upstream-nginx-module#11 can be used to preallocate list of upstreams and only use some subset of that list
2. openresty/lua-upstream-nginx-module#12 can be used for dynamic allocation of upstreams, but list can only grow, so previous PR is needed as well
3. https://twitter.com/agentzh/status/580170442150846464 — not public yet, needs manual handling of retry logic, but can replace both previous versions

Reloading nginx is not an option since it triggers graceful restart for all the worker processes. We use that mechanism with haproxy in marathoner, but long-lived sessions force previous instances of haproxy to stay alive for extended periods of time. Nginx is using more processes than haproxy, so it would be even worse. Deploying a lot can cause spinning thousands of proxying processes for no good reason.

[Tenzer]

Nginx can already do this in the Plus version: http://nginx.com/products/on-the-fly-reconfiguration/

[bobrik]

Yep, that's another option starting at $7000 annually for 5 servers.

[Tenzer]

You can also buy the license for one server at $1500 per year: http://nginx.com/products/pricing/. I'm not saying it's cheap or anything but it's an alternative if people wasn't aware of it.

[bobrik]

Another option: https://github.com/yzprofile/ngx_http_dyups_module

[bobrik]

Looks like dyups can do the trick: https://github.com/bobrik/zoidberg-nginx. Nginx config and lua scripts could give an idea how it works.

I'm not sure about stability of this thing, though.

[subnetmarco]

Just a quick update on this, this feature is important and we feel like an initial implementation should be done. It's not going to make it into 0.3.0 (currently working on implementing other major features, including SSL support and path-based routing), but it definitely should show up within the next two releases. We are trying to keep the release cycles very short, so it shouldn't take too much time.

Of course pull-requests are also welcome.

[bobrik]

Can you tell me how this is going to be implemented?

On Wednesday, May 20, 2015, Marco Palladino notifications@github.com wrote:

Just a quick update on this, this feature is important and we feel like an initial implementation should be done. It's not going to make it into 0.3.0 (currently working on implementing other major features, including SSL support and path-based routing), but it definitely should show up within the next two releases. We are trying to keep the release cycles very short, so it shouldn't take too much time.

— Reply to this email directly or view it on GitHub https://github.com/Mashape/kong/issues/157#issuecomment-103832824.

Regards, Ian Babrou http://bobrik.name http://twitter.com/ibobrik skype:i.babrou

[subnetmarco]

@bobrik As you pointed out in one of your links, apparently the creator of OpenResty is building a balancer_by_lua functionality which should do the job - so I am investigating this option.

The alternative is taking one of the existing pull requests on the lua-upstream module and contribute to them to make them acceptable and implement any missing feature we might need.

[bobrik]

@thefosk the only public info about balancer_by_lua is the tweet by @agentzh. Contributing to existing PRs to lua-upstream module also involves his approval :)

Take a look at ngx_http_dyups_module, this stuff reuses logic from nginx, as opposed to balancer_by_lua. It worked for me in my tests without crashes: 1-100 upstreams, updating every second with full gradual upstream list replace, 8k rps on 1 core with literally no lua code execution when serving user requests. Not sure about keepalive to upstreams, https and tcp upstreams, though.

[subnetmarco]

@bobrik yes, we will start working on this feature in the next releases, so we will monitor any announcement about balancer_by_lua in the meanwhile. If balancer_by_lua won't be released publicly during this time, then we will need to find another solution.

The requirement for Kong would be to dynamically create an upstream configuration from Lua, then dynamically populate the upstream object with servers and use it in the proxy_pass directive. Do you think we can invoke ngx_http_dyups_module functions directly from Lua bypassing its RESTful API?

The use case wouldn't be to update an existing upstream configuration, but to create a brand new from scratch, in pseudo-code:

set $upstream nil;
access_by_lua '
  local upstream = upstream:new()
  upstream.add_server("backend1.example.com", { weight = 5 })
  upstream.add_server("backend2.example.com:8080", { fail_timeout = 5, slow_start = 30 })
  ngx.var.upstream = upstream
';
proxy_pass http://$upstream

[bobrik]

@thefosk I create upstreams on the fly and update them on the fly with ngx_http_dyups_module. Moreover, I do it from lua code in RESTful API.

Take a look:

https://github.com/bobrik/zoidberg-nginx/blob/master/nginx.conf https://github.com/bobrik/zoidberg-nginx/blob/master/zoidberg-state-handler.lua https://github.com/bobrik/zoidberg-nginx/blob/master/zoidberg-proxy-rewrite.lua

Your pseudo-code implies that you create upstream on every request, I only do that on every upstream update. In zoidberg-nginx there is also some code for checking if upstream exists to prevent endless loops, but I found out that it is avoidable with this trick:

        location / {
            set $where where-am-i.zoidberg;
            proxy_pass http://$where;
        }

Upstream where-am-i.zoidberg is created on the fly and it's not a real domain name, no recurse proxying to itself until worker_connections are exhausted happens.

[subnetmarco]

@bobrik thank you, I will look into this

[Gingonic]

I'm also looking at an API manager that makes sense for mesos/marathon. After spending much time with my friend Google I came to the conclusion that right now there is only one option available : choose a service discovery (consul, haproxy brige, zoidberg,...) and add an API Proxy on top of it (Kong, Repose, Tyk, ApiAxle, WSO2 AM, etc.). Frankly I don't see why I should put a proxy on front of a proxy. It would make a lot of sense to have a lightweight API manager+service discovery service in one piece of middleware. So +1 for this feature. What is the planing state?

[krishnaarava]

+1 for this feature

[ngbinh]

same here :+1:

[neilalbrock]

Another +1 for me good sir

[agentzh]

The balancer_by_lua* directives from ngx_lua will get opensourced soon, in the next 3 months or so.

[subnetmarco]

@agentzh very good news, looking forward to trying it

[jdrake]

@thefosk +1

[sonicaghi]

+1

[bobrik]

@agentzh any chance to get TCP support in balancer_by_lua as well?

[agentzh]

@bobrik I'm not sure I understand that question. Are you talking about doing cosockets in the Lua code run by balancer_by_lua or you mean using balancer_by_lua in stream {} configuration blocks instead of http {} blocks?

[bobrik]

@agentzh I'm talking about using balancer_by_lua in stream blocks.

[agentzh]

@bobrik Then it's the duty of ngx_stream_lua_module while the current one is ngx_http_lua_module. Different thing :)

[bobrik]

Different subsystems, got it. Link with more info from you:

• https://groups.google.com/d/msg/openresty-en/jC6EWffdg_Q/GCdamhSbWYIJ

[geoand]

+1 I am really looking forward to this and more specifically pairing with Consul and possibly Consul-template

[thomaskvnze]

+1, this is the missing part in my CoreOS setup. I need to loadbalance request to dynamically created instances of my microservice.

[harlow]

+1

[mavencode01]

+1

[agentzh]

I just opensourced balancer_by_lua* in GitHub's balancer-by-lua branch, with the permissions from CloudFlare:

https://github.com/openresty/lua-nginx-module/tree/balancer-by-lua

It currently lacks documentation but its declarative test suite can serve this purpose (I hope):

https://github.com/openresty/lua-nginx-module/blob/balancer-by-lua/t/133-balancer.t

Please let me know if it works for you. Thank you.

[subnetmarco]

@agentzh thank you, I will take a look at it now

[thibaultcha]

Thanks @agentzh, great news!

[sonicaghi]

:beers:

[subnetmarco]

Wondering if this issue should be implemented into Kong core, or into a Plugin.

I first thought it should be into core, but then @sinzone sparked the idea that if it was a plugin, then we could enabled very powerful use-cases, like:

• Customizing the plugin per Consumer, so that for example an enterprise consumer could be proxied to a more powerful upstream server/cluster.
• Once #505 has been implemented, it could also be possible to load balance the request per endpoint, so that a very expensive endpoint (performance-wise) could be proxied to a more optimized upstream cluster
• etc

Having the Load Balancing feature as a plugin raises a couple of new question though:

• Technically it would be possible to add a new API to the system without specifying the upstream_url, because the Load Balancing plugin will inject the upstream server address every time.
• The same uses cases of the Load Balancing plugin could also be applied currently even if there is only one upstream_url. Maybe I don't want to load balance a specific consumer to a set of upstream servers, maybe I want to load balance it to only one specific upstream_url, which logically means that.....
• ....the current upstream_url could be replaced by having the Load Balancing plugin configured with just one backend URL.

I am just thinking out loud, any feedback?

[ahmadnassri]

why not both? load balancing in the core, plugin adds customization (per consumer, #505)

[orliesaurus]

How do you know if a customer is enterprise? Authorisation? Then Maybe you can put a flag there if you make it a core functionality to say "hey careful, upstream this guy to the most stable upstream url" something like route_privileges in authorisation, which then routes to the number one priority upstream url defined in the core & the kong admin can define the priority list

[sonicaghi]

@thibaultCha @Tieske what do you think?

[Floby]

I like that this feature is coming =) If I can give advice on the plugin/core debate. I like that the core of kong is really simple and fits a lot of use cases by itself with just proxying to one upstream. It reduces the "time to first api" for newcomers. So I'd say

• keep one mandatory upstream for a core API
• make the multi-upstream a plugin with its own policies about the existing 'core' upstream
• as a policy I'd imagine including the 'core' upstream as a fail-safe upstream in the upstream list.

[subnetmarco]

@Floby I agree, that seems to be the best decision, and if core keeps working as it is it will also not affect users who are not planning to use the Load Balancing feature.

[thibaultcha]

Or simply transform the current upstream_url in core in an array of upstreams which must contain at least 1 element. After all, it won't affect any user since a simple migration would do the job of migrating the old model to the new one, as always. It would also avoid creating yet one more Cassandra table (more queries to retrieve data, and more tables for Cassandra to maintain in the cluster).

[ahmadnassri]

I like @thibaultCha's suggestion better :+1:

we can still create a "multi-upstream"-like plugin if further functionality might be required beyond the scope of "load balancing"

[subnetmarco]

Both are viable solutions, one adheres more to the "everything should be a plugin" mantra, the other one extends the core.

[Floby]

I prefer things to do one thing well. Simple is most often preferable to powerful. In this case adding the multi upstream to the core forces newcomers to learn more concepts than they need to in order to get started.

When choosing an API gateway tool, I also stumbled across Tyk which is a single piece of software. It's incredibly harder to get your first API running because you have to set many upstreams, your API version, tag it for a specific environment, configure ACL and so on.

So I actually would like load balancing for my APIs =) but I think it's better for Kong as a product to have it done in a plugin

Le 8 déc. 2015 03:05, "Marco Palladino" notifications@github.com a écrit :

Both are viable solutions, one adheres more to the "everything should be a plugin" mantra, the other one extends the core.

— Reply to this email directly or view it on GitHub https://github.com/Mashape/kong/issues/157#issuecomment-162735518.

[thibaultcha]

Before taking any decision, we need a proof of concept with the balancer_by_lua directive anyways. Because of the way Kong is built, it might make more sense and be less clumsy to implement this in the core, or it might be possible to robustly implement this as a plugin, we do not know yet.

Once we have a POC, we can take arguments into account and chose an implementation.

[Floby]

Makes 100% sense

Le 8 déc. 2015 09:09, "Thibault Charbonnier" notifications@github.com a écrit :

Before taking any decision, we need a proof of concept with the balancer_by_lua directive anyways. Because of the way Kong is built, it might make more sense and be more less clumsy to implement this in the core, or it might be possible to robustly implement this as a plugin, we do now know yet.

Once we have a POC, we can take arguments into account and chose an implementation.

— Reply to this email directly or view it on GitHub https://github.com/Mashape/kong/issues/157#issuecomment-162809157.

[ahmadnassri]

@floby's point on simplicity persuaded me :)

@thibaultcha is correct in needing a POC first.

[snakelab]

:+1:

[z0mbix]

+1

[Tieske]

I've been looking into this, and it seems there is little we can reuse.

Considering that we have multiple api's to support, which must be configured with their own set of peers. Within this set we must load balance requests.

With the tools at hand, we cannot dynamically create upstreams (upstream per api). We also cannot dynamically add servers to an upstream. So if that cannot be done, then the only benefit that can be derived from using the core nginx loadbalancer must be based on settings that can be used on a per request basis. This leaves only the set_more_tries function in the new balancer_by_luacontext. But if we use it (with only one upstream), we can only set the reasons for failure globally in the nginx config file (whether something failed is determined by nginx). But we weant to be able to specify failure reasons per api.

Conclusion: we must rebuild in Lua, cannot use any of the existing stuff. Except the healthcheck module, which can be reused with minor adjustments.

Am I missing something? thoughts?