nginx proxy on front of paz?

[hyperbolic2346]

I have a historical nginx setup, which proxies all my servers. What I do is publish into etcd and I have confd watching that and writing out my nginx config file. I do this to keep requests to certain services locked to internal access only and other services are public. I'm thinking that this matches in a way haproxy, but just as a stop-gap until I convert over I was planning on using nginx in front of paz(haproxy).

This seems to work, but I do see some issues. The first is that occasionally the page refresh fails and the second issue is that the services tab just errors. Looking through the network requests I was able to find that I needed to expose paz-web, paz-orchestrator, and paz-orchestrator-socket. I also found that I needed to pass websocket connections with

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;

But I'm not sure where to being to find out why things are still failing.

Also, please advise if it would just be easier to convert my services to haproxy. I'm not against that at all. I am concerned about the availability of haproxy, but I assume I can add in some ip restrictions for the proxy sites?

[lynchc]

+1. I'm using nginx as well

[lukebond]

Something that consuses me here is that I don't understand how you can be using HAProxy and nginx together, given that HAProxy is binding itself to port 80 on each CoreOS host. Do you have nginx elsewhere proxying to Paz rather then on the same box, or do you have one of them on something other than port 80?

That aside, the issues you're seeing I assume come down to the fact that the HAProxy "plumbing" is all based on HTTP host header, so if requests come in (perhaps through nginx) and they don't have this prefix in the HTTP host header, then things will fail. And yes, what is needed to be routable from the outside is paz-web.*, paz-orchestrator.*, paz-orchestrator-socket.* and paz-scheduler.* (the latter as of very recently).

Btw @hyperbolic2346 your re-writing of nginx config sounds cool, and as you say, similar to Paz's HAProxy setup. I've no time (nor sufficient knowledge of nginx) to do so, but if people would find it useful to use nginx instead of HAProxy to do the multi-host Docker networking then I'd be interested if someone wanted to build it and provide the option to Paz users.

Finally, we're considering some fairly wide-ranging changes to these parts of Paz that may affect all this. See #33.

[hyperbolic2346]

@lukebond Yes, I'm running haproxy on a different port on my setup to keep nginx going on port 80. I know that it is redundant at best to run both, but as I said it was for transition. I'm not sure which I would like to use in the long run as I have never used haproxy. I'll read up on it and see what I want to do. I'm sure the problem is something that isn't properly being forwarded over.

Looking at how haproxy is configured, it seems like it is pretty much exactly how I am doing nginx. Using confd to write a config from etcd and restart. It should be very simply to drop my own template via a docker volume and have haproxy behave how I would like.

[lukebond]

As far as I can tell that would be the simplest solution. Some advantages of HAProxy (bearing in mind I've little experience with nginx) is that the config reload can be done without a restart, and we can also drain off connections as we take services out of backends to give us zero downtime deployments (not currently being capitalised on).

Also if your current system does TCP and not just HTTP then you'd be losing functionality (See https://github.com/paz-sh/paz-haproxy/issues/3)

[hyperbolic2346]

@lukebond You've got me looking at weave now. If I'm going to transition everything to something new it might be that. It looks very promising even for my own container communication. I currently have a ton of etcd data that would simply not be needed with weave. No promises, but you might end up with a pr for weave. I'm going to concentrate on that stuff instead of nginx or haproxy.

[lukebond]

No promises, but you might end up with a pr for weave.

<3