Dynamically create and remove upstreams?

[josh-richardson]

Hi,

First of all, this is a super awesome project & exactly what I've been looking for in the last few weeks - it was a difficult one to find!

At the moment, as far as I can see, dshackle has support for a static list of upstreams as defined in a yaml file, but there's no way to remove upstreams or add them depending on application load/etc. Would such a feature be beneficial to the project?

If so, I'd be quite interested in implementing.

[splix]

Hi @josh-richardson

Thank you, I'm glad you find it helpful.

It's indeed an interesting feature, and I was thinking about it from the beginning. Though it's not available at this moment, but some design decisions were made considering this as a possible feature. So it should be possible to add. Adding it though, would require many changes to different places of the app code. I don't have immediate plans to add that feature, but would appreciate if you can implement it. I'd be happy to answer questions about how it could be added, and guide through the code.

[flokli]

It's already possible to pass DNS names at connection.* in cluster.upstreams[] in the config.

What if dshackle would simply (periodically, according to TTL) resolve that name, and in case multiple records are returned, register each of them as upstreams?

Of course, it'd also need to take them out in case they disappear from the list.

This should tie in nicely into various service discovery mechanisms (such as k8s headless services, Consul DNS, …).

[splix]

That could be an option too. And probably easier to implement than other ways of dynamic upstreams.