Add support for DNS Dyamic Update

[ConnorDoyle]

Dynamic update is a feature that allows additional services to be registered with a central DNS server. Its role in a DNS-based service discovery scheme is summarized well in IETF RFC 6763, Appendix A.

In short, this would allow Mesos-DNS to be useful for discovering non-Mesos services, at little cost, and therefore become more valuable as a generalized service discovery tool.

At the same time, the Mesos master should remain the source of truth for records in the managed namespace.

Related to #132.

[kozyraki]

@ConnorDoyle and @stuart-warren The challenge with this is the following: Mesos-DNS does not have persistent store on its own. So, it may crash at any point losing all its state. For state received from the Mesos-master, we can just hit the master again and get it. But for state provided externally, this is not as simple. Moreover, it is not clear who will keep an eye on mesos-dns and notify others to send it the dynamic updates again.

@jdef is working on a plugin model so that K8S can insert records for K8S services into Mesos-DNS. In that case, K8S is responsible for persisting this info.

[tarnfeld]

Trying to mention in the most non-pluggy way but @ConnorDoyle we actually wrote and run our own fully fledged DNS nameserver that uses etcd as a backing store, and it's itchingly close to having full support for those dynamic updates, there's a branch you can try out that's very nearly finished.

https://github.com/duedil-ltd/discodns/pull/31

This project might fit more naturally into your stack if you're using mesos/zookeeper/etcd/coreos/docker etcetcetc as it can provide stateless nameservers but with a very reliable backing store.

For what it's worth, I wouldn't say that mesos-dns should be responsible for consuming and managing these non-mesos entries in the name system. DNS is designed to be a namespace built up of many different authorities, and you should probably look at fronting mesos-dns with something like BIND9 or Unbound if you want to have better support for different types of names. I'd very much like to see mesos-dns stay true to provided only dns records for a mesos cluster, and remaining as simple and stateless as it currently is.

^ We're doing exactly this and have been in production for almost a year. Turns out it works really well :smile:

Note: The reasons outlined in that appendix are precisely why we rolled this model out over a year ago, and have been reaping the benefits ever since.

[ConnorDoyle]

@tarnfeld and @kozyraki great points! Dynamic updates could be handled by a separate solution. Closing this for now, this will be a good reference for the next time the question comes up.

[kozyraki]

@tarnfeld when we started mesos-dns we struggled a lot with whether we should write our own (very simple) tool or use consul, skydns, etc (I will admit I did not know about discodns at the time, maybe you should be more pluggy about it :). The main reason we wrote our own is that we did not want to force Mesos users install etcd/raft/a gossip algorithm/... in addition to Zookeeper to make their system work. I have nothing against any of these individually, but I know very few people that want to deal with 2 of them at scale. So, we went as simple as possible. Hopefully, the Mesos state abstraction will improve over time and we can do a few more interesting thing, but in general I am with you: Mesos-DNS should solve one problem well and let others deal with other aspects.

So, the MVP question is: can discodns and mesos-dns play along nicely if someone needs bother? Are we missing some features to make it work?

[tsenart]

What are the data consistency requirements mesos-dns records? If we'd run multiple instances of mesos-dns, would it be OK to sacrifice consistency for availability?

I'm having some ideas of an eventually consistent data replication scheme based on CRDTs and local file storage. :-)

[tsenart]

/cc @kozyraki