spec: use TXT dns records for discovery

[philips]

From @philips on December 8, 2014 21:3

@blalor said "I feel like there should be a DNS component to this, as well. Perhaps a TXT record?" in #210.

Lets discuss that topic here.

/cc @geekgonecrazy

Copied from original issue: coreos/rocket#242

[philips]

From @geekgonecrazy on December 8, 2014 21:24

For me there are a few reasons for a DNS record.

• I prefer to keep un-needed tags out of my website markup.
• I compile my website / web app. So my sites markup is actually in a docker container currently and versioned. So i'd have to bake it in there.
• I can update my DNS via a REST api.
• Already partially using DNS for service discovery.
• Running internal DNS for intranet. Could keep info completely private with out having to maintain 2 versions of my site, or have to use a different domain.

[philips]

From @blalor on December 8, 2014 21:25

All of those things. Said far better than I could do it. :-)

[philips]

@geekgonecrazy @blalor Can you provide some example TXT record formats to consider for this? I don't have a good first-hand experience with the constraints or features of the format.

[philips]

From @robszumski on December 8, 2014 21:39

TXT record would also mean you're not spinning up extra infrastructure just to serve a few meta tags on a certain domain.

[philips]

From @geekgonecrazy on December 8, 2014 21:42

I'm in no way qualified to give a definitive answer. But here's what my google-fu gives me.

From RFC1464

The general syntax is:

        <owner> <class> <ttl> TXT "<attribute name>=<attribute value>"

From reading the App container spec i'd guess something like this:

ac-discovery content="prefix-match url-tmpl" ac-discovery-pubkeys content="prefix-match url"

or

ac-discovery="prefix-match url-tmpl" ac-discovery-pubkeys="prefix-match url"

Records could be separate or in the same record. If i'm not mistaken you're limited to 255 characters.

If you wanted to squeeze them all in the same record could do:

app-container discovery="prefix-match url-tmpl" discovery-pubkeys="prefix-match url"

[philips]

@miekg You are a DNS expert, any suggestions on resources to help us design a TXT record format. Essentially we want to have the same thing as these HTTP meta tags only for TXT records:

https://github.com/coreos/rocket/blob/master/app-container/SPEC.md#meta-discovery

Any help would be greatly appreciated.

[philips]

From @geekgonecrazy on December 8, 2014 22:26

Just realized i'm using " inside the example records. Looks like they'd have to be escaped which would be messy. So would have to break it up a bit.

ac-discovery: example.com. 730 IN TXT "ac-discovery prefix=example.com url-tmpl=https://example.com/{os}/{arch}/{name}-{version}.{ext}?torrent"

ac-discovery-pubkeys: example.com. 730 IN TXT "ac-discovery-pubkeys prefix=example.com url=https://example.com/pubkeys.gpg"

But yeah a DNS expert would know.

[philips]

From @miekg on December 8, 2014 22:31

Just pulled in, haven't read the spec. This is all new, so a client needs to understand the records? If so, don't, DON'T use a TXT record, invent a new type (it's easy and the best thing to do - I can help). You can copy the TXT semantics but they are not great. If you only need two elements, just define the new RR to do so.

[philips]

From @geekgonecrazy on December 8, 2014 22:42

don't, DON'T use a TXT record, invent a new type (it's easy and the best thing to do - I can help)

@miekg Correct me if i'm wrong.. But wouldn't this require us to be running our own DNS servers so that we could then add this new type in?

If so, that'd defeat the usefulness of using a DNS record, right?

[philips]

From @miekg on December 8, 2014 22:46

Not per se, as all dns servers should support unknown records (but yes some don't support those) On 8 Dec 2014 22:42, "Aaron Ogle" notifications@github.com wrote:

don't, DON'T use a TXT record, invent a new type (it's easy and the best thing to do - I can help)

@miekg https://github.com/miekg Correct me if i'm wrong.. But wouldn't this require us to be running our own DNS servers so that we could then add this new type in?

If so kinda defeats the usefulness of using a DNS record, right?

— Reply to this email directly or view it on GitHub https://github.com/coreos/rocket/issues/242#issuecomment-66202091.

[philips]

From @geekgonecrazy on December 8, 2014 23:5

If this is to be added to part of the App Container spec i'd think we would want to only use established DNS records. So that it could be used regardless of the DNS server used.

I don't think i've personally ever seen one that would allow you to feed an unknown record type. Other then maybe a bind9 server. Definitely not something like Amazon's Route 53 or any other DNS service.

[philips]

From @blalor on December 8, 2014 23:10

Yeah, that's why I suggested TXT: universality. Unless there's another suitable record type. IANADNSE. (I am not a DNS expert)

[philips]

From @miekg on December 9, 2014 9:9

Also see http://miek.nl/posts/2013/Oct/12/What%20to%20put%20in%20the%20DNS/

TXT can work, but when requesting the record you can all TXT records under that name, so maybe it is prudent to use a special sub domain? Also just put base64 data in the record if you're scared of encoding issues (which will be plenty full). Just make sure that you split the fragments on the 255 per-text limit boundery.

I will check the spec and send a longer reply this evening (GMT)

[philips]

From @blalor on December 9, 2014 11:41

I do think an ACI-specific subdomain for the TXT record would be fine; the record would not have to be a resolvable name (ie one with A or CNAME records). I don’t know how trustworthy this source is, but it does state that an underscore can be used with TXT subdomains: http://domainkeys.sourceforge.net/underscore.html http://domainkeys.sourceforge.net/underscore.html

So you could have a single _ac.example.com http://ac.example.com/ record, or multiples to support future keys without hitting the 255 byte limit. Say _discovery._ac.example.com http://ac.example.com/ and _discovery-pubkeys._ac.example.com http://ac.example.com/.=

[philips]

From @miekg on December 9, 2014 11:53

Yes you can use underscores in names. What 255 limit? You can store 16k in a txt record. On 9 Dec 2014 11:41, "Brian Lalor" notifications@github.com wrote:

I do think an ACI-specific subdomain for the TXT record would be fine; the record would not have to be a resolvable name (ie one with A or CNAME records). I don’t know how trustworthy this source is, but it does state that an underscore can be used with TXT subdomains: http://domainkeys.sourceforge.net/underscore.html < http://domainkeys.sourceforge.net/underscore.html>

So you could have a single _ac.example.com http://ac.example.com/ record, or multiples to support future keys without hitting the 255 byte limit. Say _discovery._ac.example.com http://ac.example.com/ and _discovery-pubkeys._ac.example.com http://ac.example.com/.=

— Reply to this email directly or view it on GitHub https://github.com/coreos/rocket/issues/242#issuecomment-66270501.

[philips]

From @blalor on December 9, 2014 11:55

On Dec 9, 2014, at 6:53 AM, Miek Gieben notifications@github.com wrote:

Yes you can use underscores in names. What 255 limit? You can store 16k in a txt record.

Ah, I thought someone else said that was the limit. Why not just an _ac-discovery.example.com http://ac-discovery.example.com/ with a JSON blob?=

[philips]

From @miekg on December 9, 2014 12:25

That can work, but do note that due to the silliness of the txt record, this needs to be split in 255 bytes chunks.

Still think a new RR makes most sense, but that can be a parallel effort. On 9 Dec 2014 11:55, "Brian Lalor" notifications@github.com wrote:

On Dec 9, 2014, at 6:53 AM, Miek Gieben notifications@github.com wrote:

Yes you can use underscores in names. What 255 limit? You can store 16k in a txt record.

Ah, I thought someone else said that was the limit. Why not just an _ ac-discovery.example.com http://ac-discovery.example.com/ with a JSON blob?=

— Reply to this email directly or view it on GitHub https://github.com/coreos/rocket/issues/242#issuecomment-66271974.

[philips]

From @blalor on December 9, 2014 12:37

Ok, and that would be the 255 byte limit…=

[philips]

From @miekg on December 9, 2014 13:3

You can store 16 k but each segment in a txt can only hold 255 bytes because it is preceded by a one byte length. There can be multiple segments in a single Txt record.

Txt records are complete crap. On 9 Dec 2014 12:37, "Brian Lalor" notifications@github.com wrote:

Ok, and that would be the 255 byte limit…=

— Reply to this email directly or view it on GitHub https://github.com/coreos/rocket/issues/242#issuecomment-66276404.

[philips]

From @geekgonecrazy on December 9, 2014 18:31

Why all the need for encoding or json? Seems like we might be overcomplicating things. Should be super simple.

ac-discovery

TXT record with subdomain _ac-discovery With value: "prefix=example.com url-tmpl=https://example.com/{os}/{arch}/{name}-{version}.{ext}?torrent"

_ac-discovery.example.com. 1800 IN  TXT "prefix=example.com url-tmpl=https://example.com/{os}/{arch}/{name}-{version}.{ext}?torrent"
_ac-discovery.example.com. 1800 IN  TXT "prefix=example2.com url-tmpl=https://example.com/{os}/{arch}/{name}-{version}.{ext}?torrent"

ac-discovery-pubkeys

TXT record with subdomain _ac-discovery-pubkeys With value: "prefix=example.com url=https://example.com/pubkeys.gpg"

_ac-discovery-pubkeys.example.com. 1800 IN TXT  "prefix=example.com url=https://example.com/pubkeys.gpg"

Very similar to the meta tags, very easy to write, very easy to parse.

Grab txt records for subdomain _ac-discovery parse the records returned. With TXT you can do multiple records for that same subdomain same as with meta tags.

[blalor]

The reason I dared to suggest JSON is that the parsing's already taken care of for you. :-) I like the two-record solution @geekgonecrazy showed above; simple, extensible.

[geekgonecrazy]

@blalor yeah JSON would be easier to parse. But with the simple key/value pair its fairly easy to split on space to get the pairs, then on the '=' to get the key and value. Not super elegant but simple.

Started writing up a simple deal in go earlier today to make sure it was as easy to do as I thought. Actually using @miekg 's dns package. Had a couple of meetings, so wasn't able to finish.

[dapithor]

I am naive to what exactly you need to have in the discover mechanism/medium but my first thought over multiple DNS TXT entries would be 2 entry DNS record types.

ac-discovery DNS SRV record:

DNS SRV record and DNS TXT record (_service._proto.name. TTL class SRV priority weight port target.)

Example: _{name}-{version}-{os}-{arch}._aci-discovery.example.com. 86400 IN SRV 0 5 80 {name}-{version}-{os}-{arch}.example.com.

[blalor]

@dapithor that would be an abuse SRV spec, which could cause problems. SRV targets are supposed to be names of A records (which is overly prescriptive, in my opinion). I tend to put CNAMEs in there, but I've only done that with Route53. To @geekgonecrazy's point I think blatant DNS RR abuse should be avoided for best compatibility with DNS providers.

[geekgonecrazy]

@dapithor Yeah take a look at RFC2782

Doing it this way would require you add a new dns record for every release made. ;-)

@philips what was the thinking behind having prefix?

<meta name="ac-discovery" content="prefix-match url-tmpl">
<meta name="ac-discovery-pubkeys" content="prefix-match url">

Are you thinking of being able to specify a registry and have one handle multiple? This is what I suspect but just wanted to verify.

[bacongobbler]

@geekgonecrazy the inspiration comes from https://golang.org/cmd/go/#hdr-Remote_import_paths, which uses the prefix as verification that the prefix has the same tag.

[geekgonecrazy]

@bacongobbler ah.. so its purely verification. Nothing to do with being able to host multiple prefixes on the same domain. Still I suppose it could be used that way.

[miekg]

Yes https://github.com/appc/spec/issues/3#issuecomment-66342583, would work perfectly well.

[philips]

@miekg So you are ok with https://github.com/appc/spec/issues/3#issuecomment-66342583 using TXT records?

[miekg]

[ Quoting notifications@github.com in "Re: [spec] appc: use TXT dns record..." ]

@miekg So you are ok with https://github.com/appc/spec/issues/3#issuecomment-66342583 using TXT records?

Yes. And I'm willing to make the effort and get you an RR type? (It's not that hard).

/Miek

Miek Gieben

[geekgonecrazy]

https://gist.github.com/geekgonecrazy/eef4f2fa720d4d349176 is what I got with a few minutes of playing around with it yesterday. I'm a golang noob still, so probably easier ways of doing it. :)

[dapithor]

Not trying to bump an old item but this is somewhat how I was thinking to do this with SRV (per my previous comments). I didn't articulate it well enough though... not sure how close you all work with mesos but for reference: http://mesosphere.github.io/mesos-dns/docs/naming.html (which I see uses miekg's framework).

[philips]

@dapithor I don't understand how the mesos-dns tool helps here. Can you explain a bit further?

[geekgonecrazy]

@dapithor Maybe try explaining again?

Because from what I see they are using the SRV records to point to an ip. Which is what's supposed to be in an SRV record.

The problem with just an ip address is its still not telling rocket(or any tool in the future) where to grab the .aci file. Its just telling it which host its on. Since its fetching via http, it needs to be able to determine the full url.

[cdaylward]

I think there are two things going on here:

1. Should a DNS discovery mechanism be added to the spec?
2. Should the discovery process be as prescriptive as it is? I really like the two discovery mechanisms as they are currently outlined, however I would like to see discovery reframed as "pluggable, ordered discovery strategies". Simple discovery is a strategy. Meta-discovery is a strategy. DNS discovery is a strategy. I believe it would be beneficial to outline their structure but not prescribe which ones must be used or in which order. This is a little OT here, should this be a separate discussion thread?

As far as using DNS goes, SRV is I believe half the story. Combining an NAPTR record with the use of SRV (or without) gives you resource location and URI rewrite rules. This could be conceivably used to turn label sets into resource locators.

[jonboulle]

2) is a distinct but good point that I would also like to address, mind filing another issue/PR for that?

[geekgonecrazy]

@cdaylward I agree, they should probably be pluggable. Order of discovery methods would probably be an entirely separate discussion.

How common are NAPTR records? I honestly can't say i've ever even seen that record type until now. I checked both DNS providers i've used in the past, and neither of them even offer that record type.

Looks intriguing though..

[cdaylward]

@jonboulle Not at all. Will do.

[cdaylward]

@geekgonecrazy NAPTR records are used extensively by VOIP carriers, their peers, and clients (I know this first-hand). I've also used them on other internal networks for other types of resource location (you can use it to select based on preferred protocol). The RFC has examples of general use, e.g. (this one does not use the rewrite field)

 IN NAPTR 100  50  "s"  "z3950+I2L+I2C"     ""  _z3950._tcp.gatech.edu.
 IN NAPTR 100  50  "s"  "rcds+I2C"          ""  _rcds._udp.gatech.edu.
 IN NAPTR 100  50  "s"  "http+I2L+I2C+I2R"  ""  _http._tcp.gatech.edu.

[miekg]

[ Quoting notifications@github.com in "Re: [spec] appc: use TXT dns record..." ]

@geekgonecrazy NAPTR records are used extensively by VOIP carriers, their peers, and clients (I know this first-hand). I've also used them on other internal networks for other types of resource location (you can use it to select based on preferred protocol). The RFC has examples of general use, e.g. (this one does not use the rewrite field)

IN NAPTR 100  50  "s"  "z3950+I2L+I2C"     ""  _z3950._tcp.gatech.edu.
IN NAPTR 100  50  "s"  "rcds+I2C"          ""  _rcds._udp.gatech.edu.
IN NAPTR 100  50  "s"  "http+I2L+I2C+I2R"  ""  _http._tcp.gatech.edu.

Indeed NAPTR is just a normal record, except nobody uses it on the internet. I don't think DNS server providers will allow you to add NAPTR record out of the box.

As I said before, if you going to take the route of service discovery in the DNS you probably best of designing your own record type. (To hell with those pesky DNS service providers). And ... maybe al fallback to TXT (if you must).

/Miek

Miek Gieben

[dapithor]

Per @cdaylward mentioned that was my initial thoughts too. Using a SRV record doesn't fulfill the request completely but it does provide purpose of listing a host, port, and protocol for a service of ACI's. The second part I mentioned was to have a default location and if needed a corresponding TXT entry. Of course I like the idea of a custom Resource Record per miekg's advice but to keep things "simple" and able to be used on most platforms SRV for service info and TXT for config if needed. IMO having a custom RR you might as well just have a custom tool which is full circle to etcd like usage. This allows an instance to run on a different port and maybe in future different protocol requirements and still leave the config alone (TXT record) unless no other port except 443 will be used or always doing an URI://location:port/{opt1}{opt2}{etc} is preferred then I say SRV is useless.

[miekg]

How is "lookup SRV" and then "lookup TXT" not custom? I'm not sure what the requirements here are?

The only thing going for SRV and TXT lookups is that the resource records are well known. But doing combined SRV/TXT lookups to make stuff work also requires custom code, because no resolver will do this out of the box.

[dapithor]

Fair points but application logic needs to happen regardless if one record or N records. So, if you look up anything logic needs to be in the app but SRV/TXT etc being built into most DNS servers/services alleviates the need to also have a custom dns server for a custom RR. Even with zeroconf/dns-sd (which is basically what I am advocating and should have clarified) you still need a logic/library and tell it to use it? Am I too far off base ? My thoughts were basically DNS-SD which is SRV/TXT in this wiki: http://en.wikipedia.org/wiki/Zero-configuration_networking#DNS-based_service_discovery and this RFC: http://tools.ietf.org/html/rfc6763 @philips @miekg

[geekgonecrazy]

@dapithor ah! I think I get it now. Seems you're thinking more along the line of discovery as in discovering a server that is hosting containers?

From my understanding the objective is to operate like golang package discovery works. It doesn't need to discover the server. Because the server is part of the path you enter. So you'd enter something like rkt fetch example.com/mycontainer It then tries to determine where the aci file is located using meta tags if they are available or as this proposes using a dns record if available.

As far as it discovering a registry as it seems you're proposing.. That might be a completely different matter all together.

Am I understanding this correctly?

[dapithor]

I was thinking dual fold purpose allowing you to fetch an ACI without knowing storage location but the app URI:// name because if you know the URI and basic template default of {name}-{version}-{os}-{arch}.aci and that fails the SPEC says to move onto meta discovery (using logic to parse meta-tags). My thoughts were that you just know you need [reduce-worker] and have no clue if it's on https, hdfs, sftp,etc but you can not only find out who out there is serving it and on what port and protocol but then correlate that with meta tags. So, instead of html it would be in a TXT record.

rkt fetch example.com/reduce-worker

1. check https://{name}-{version}-{os}-{arch}.aci
2. Success GOTO fetch sig ELSE start ac meta discovery [line 3]
3. DNS request a SRV record for ACI service (get back a protocol, port, location)
4. DNS request a TXT for key value pair's
5. pull down ACI with your ACE app based on SRV and TXT data

DNS SRV:

_ac-discovery._hdfs.example.com 86400 IN SRV 0 5 50070 dfs.example.com.

DNS TXT:

name="ac-discovery" content="example.com {os}/{arch}/{name}-{version}.{ext}" name="ac-discovery" content="example.com {name}-{version}-{os}-{arch}.{ext}"

Output producing:

hdfs://dfs.example.com:50070/{os}/{arch}/{name}-{version}.{ext}

I understand this might be forward thinking or not needed (adding in a SRV) but my thought was having to always modify an html page of meta tags can be cumbersome versus if you move from foo.com to bar.com or change a port you don't have to go and change those html files independently or use a configuration management template to do so. Sorry if I made this more complicated than needed to be!

[miekg]

[ Quoting notifications@github.com in "Re: [spec] appc: use TXT dns record..." ]

Fair points but application logic needs to happen regardless if one record or N records. So, if you look up anything logic needs to be in the app but SRV/TXT etc being built into most DNS servers/services alleviates the need to also have a custom dns server for a custom RR. Even with zeroconf/dns-sd (which is basically what I am advocating and should have clarified) you still need a logic/library and tell it to use it? Am I too far off base ? My thoughts were basically DNS-SD which is SRV/TXT in this wiki: http://en.wikipedia.org/wiki/Zero-configuration_networking#DNS-based_service_discovery and this RFC: http://tools.ietf.org/html/rfc6763 @philips @miekg

DNS-SD needs client support as well, but maybe more widely deployed.

[rsertelon]

@dapithor I like your idea of using SRV + TXT.

A few things come to my mind:

1/ I would suggest that TXT records should be optional, in the sense that if none is specified a default lookup format should be used (as it is done for CardDAV/CalDAV). I believe that it is {name}-{version}-{os}-{arch}.aci that is specified for now.

2/ What if the procotol is _tcp? Should HTTP(s) be used by default? Should the client use HTTP(s) based on the specified port? And should the protocol be changed ? This would force the client to check for different protocols, each should then be specified...

[b]

1) Storing this sort of data in TXT records happens frequently. For example, that's how SPF works. While a single TXT record is limited to 255 bytes of data, you can have multiple TXT records for a given label which can then be stitched together by clients. Keep in mind that any response over 512 bytes will cause DNS servers to truncate responses, making clients connect over TCP to ask the same query again, so best to keep things short.

2) Multiple lookups for service discovery is normal. MX, SRV, and CNAME records all require multiple resolutions to get from a name to an IP. The behavior is controlled by the clients, not the resolver. I would expect most of the lookups involved here would be in cache and this isn't an interactive scenario where a few extra milliseconds of DNS resolution is a concern.

3) The _proto field in an SRV record is exactly that: the protocol. In everything being discussed, that is TCP. Similarly, the _service field would be HTTP for what is being discussed. Distinguishing between HTTP and HTTPS would be down to the URI, not part of the SRV record. There is NO reason for a new RR type and going down that path would almost certainly stop anyone from deploying it.

4) DNS-SD, which actually uses 3 RR types, covers a lot of this, as has already been discussed. The exact data to be included in each record, the structure of names, the way names should be encoded and resolution performed needs to be defined as part of the app container spec since the DNS-SD specification leaves a lot of room for variation. I strongly support this standards-based approach.

[miekg]

Note that TXT records are not limited to 255 bytes, you can put 64K of data in a TXT record. Also note that you can easily have messages larger than 512 bytes with EDNS0 bufsize, the only downside of large packets messages be UDP fragmentation

Usually a server is smart in that it will try to put the IP address for MX, SRV in the additional section of a message. In the case of CNAME, the target (if the servers has it) is put in the answer section.

There is all the reason for a new RR type, because that is the easy thing to do. Reusing TXT is just about the dumbest thing to do here. BUT getting a supported everything is a pain. I agree with that.

[b]

I understand how quickly these sorts of discussions devolve into protocol pedantry, but here we go. A given TXT RR in a zone file is limited to 255 bytes. A label may have multiple TXT RRs, totaling up to 64K. EDNS0 is not universally supported, so 512 bytes is the conservative choice certain to work as expected in all situations. For example, there are still many firewalls deployed that happily discard UDP DNS responses larger than 512 bytes.

While servers might add data in the additional section, that is an optimization the number of round trips. It is still on the client, not the resolver, to pick out the records it needs and what to do with them. There is no reason servers couldn't or wouldn't piggyback the TXT records being discussed, just as they do with SPF and with things I, and no doubt many others, have implemented. It's not "dumb", it's using the tools as intended.

Asserting that there is a reason is not the same as demonstrating there is a reason. There is no suggestion the desired behavior cannot be achieved with existing RR types, only that some slight optimization in resolution might be possible at the cost of incompatibility with all deployed DNS implementations. SPF and DNS-SD are just two of many examples of how to properly use existing resource records in combination to achieve specific results and there is no reason this situation is any different.