search

varnish / varnish-modules

Collection of Varnish Cache modules (vmods) by Varnish Software
Other
184 stars 86 forks source link

Consider adding ${DYNAMIC_BACKEND_VMOD} #4

Closed lkarsten closed 8 years ago

lkarsten commented 8 years ago

Guillaume has written a vmod that allows for creating fresh backends per request. This includes doing the DNS lookup, which has been on many people's wishlist for many years.

https://github.com/gquintard/libvmod-gwist

daghf commented 8 years ago

Discussion in #5

Consensus from last Monday seems to be that vmod-named is the main candidate, and Dridi & Guillaume should fight it out between themselves to get the possibly missing functionality in there.

dridi commented 8 years ago

There's currently no real overlap between vmod-named and vmod-gwist and one thing that @gquintard needs is having an arbitrary port during backend selection.

I don't intend to do that for several reasons, including complexity and diverging too much from Varnish 3's DNS director. Also talking about the port is not accurate, it's actually a service name. One thing I want to test is declaring services (SRV records) and confirm that you can have backends listening to different ports as long as they are properly registered.

The reason why I haven't done that (yet) is that I want to make it turnkey/reproducible and easily "try-it-at-home" demonstrable (something to show during meetups/summits or on the blog). The problem lies in the easily demonstrable requirement :(

gquintard commented 8 years ago

Caring about diverging from v3 is wrong IMO, we should care about what makes sense now and in the future.

And talking about port is accurate, I want to be able to use a host header with colon (eg: "foo.com:6565"). SRV records seem to impose a pretty big dependency on DNS (and work on the admin side) to enable something that should be easy. I don't get the complexity argument, there's an extra check to match ports, but that's about it.

Question is, what will "the world" prefer? Inclusion in the varnish modules must be driven by real-world needs.

dridi commented 8 years ago

Caring about diverging from v3 is wrong IMO [...] SRV records seem to impose a pretty big dependency on DNS

Not from v3, from v3's DNS director and to be fair the dependency on DNS is in the name. And (no pun intended but I really picked the wrong name for that) but vmod-named gets its name from the N in DNS.

I don't get the complexity argument, there's an extra check to match ports, but that's about it.

Sure, it's trivial to implement. I enjoy squashing trivial stuff once in a while.

What I want to avoid is overloading (or overriding in this case) because that tends to be confusing. I remember a discussion on "objects" where two people were talking about 2 different things until one had the good idea to do some disambiguation (and the other went "ooooh, that kind of objects"). Without thinking I can find at least 4 things called objects in Varnish.

If DNS can do it, why reinventing it in vmod-named? Isn't it a case for vmod-gwist instead?

People not willing to maintain DNS records should probably not rely on a DNS director in the first place.

rezan commented 8 years ago

DNS is just 1 usecase for dynamic backends. The other big usecase is supporting the Host header. The usecase here is the requestor defines the backend via a host and port. No upfront VCL configuration required. This is a design goal of several products.

I think its best to have 1 VMOD support both. Would you be open to a PR with this functionality? If so, lets go ahead and move forward with vmod-named and a PR patching in what we need. If we cannot support both, then lets hit the drawing board and start over.

rezan commented 8 years ago

Just to clarify, the reason why its best these are combined is because Host header routing (gwist) needs proper DNS support.

huayra commented 8 years ago

Forked the last revision from Dridi's back to our org: https://github.com/varnish/libvmod-named-1

Teaching this VMOD to do the gwist thing seems to be a trivial task. One that will make customers and users happy. So let's see what comes up from that and then eventually merge the whole thing to this repo?

wouterhummelink commented 8 years ago

Extending this to optionally use SRV records would be a great bonus since discovery tools like Consul expose service records in a port and hostname form. Health probes would not be required in this use case since Consul should only return records for alive backends anyway.

lkarsten commented 8 years ago

Hi all. With my maintainer hat on, I'm going to close this issue without a resolution. It has been open for quite a while, we can't make up our minds to get it completed and there are (for us) bigger fish to fry.