recursive/dnsdist direct insertion of objects into packetcache

[johnhtodd]

• Program: Recursor, dnsdist
• Issue type: Feature request

Short description

It would be useful to insert objects into the packetcache directly for both the recursive resolver and dnsdist without having to follow a normal recursive chain or handoff

Usecase

In locations where we have clusters of servers in a single POP which are not configured in a cascading structure it is the case that we have multiple resolvers querying origin auth servers for the same data. We would like to have the option of creating external analysis code that evaluates query streams and then injects results into resolver chains (dnsdist/pdns recursor) without actually performing the query on each individual chain. This can work cross-server within a POP, or cross-POP in circumstances where we might want to pre-load caches with new object data to improve speed or circumvent network outage conditions.

Description

In some circumstances, being able to inject a record into the packetcache would be able to save considerable round-trip searches, or would be able to circumvent failure conditions in a POP, or would be able to distribute an object between semi-independent clusters of servers within a POP without incurring additional overhead of "normal" query/responses. This might be through an API call that supports multiple insertions at once. It would seem that for completeness a command implemented at the same time for deletion of cache items may also be useful. Caches should be select-able specifically, but it may also be useful to have a method that would update all caches at the same time. dnsdist is the more important location for this to be done in our opinion, but applying the same method in both the recursor and dnsdist seems appropriate as they both have packetcache structures.
It is understood that this would require an external framework to decide what objects to distribute, and that the objects would have to be highly "trusted" within the location or network and be pre-validated by other resolvers if DNSSEC was involved. This is not seen as a problem since cache insertion in this fashion is inside the trust circle of the operator of the recursive structure.

Discussed at low resolution on #powerdns on 2019/07/05 - ticket requested by rgacogne

[johnhtodd]

I am reminded that deleting from packetcache is possible already; totally forgot about PacketCache:expungeByName() - not sure if that works on Servfail items, but I can test.

[johnhtodd]

Still interested in seeing if this might be possible.

Thinking about it a bit more, an abilty to insert into the packet cache may be quite useful for other purposes that are not obviously connected to the original idea (which may be solved by DNSQuestion:setRestartable in 1.8) The idea of shared cache is implemented via redis in unbound, but I'm not convinced that is ideal (though I like the idea a lot.) I am always happier if each system is "standalone" - shared dependencies are scary. However, a suitably flexible cache insertion model may allow a mesh to be built for local instances with only a slight amount of network interaction. If the end results of one dnsdist packet cache operation can be transmitted (unicast, even) to one or more other dnsdist receivers and inserted via this cache insertion method, then it would "magically" populate the packet cache of a dnsdist system even if it was not directly receiving queries. After a period of time of cache fill, the dnsdist instance can be added to the active pool and it would be mostly warm. This isn't "shared cache" - it is more like "cache pre-warming" or "cache broadcast". Of course, multicast would be the best way to do this, but I'd be perfectly happy with unicast as long as multiple receivers/senders were supported.

This is way, way better than using "tee" to do the same thing because the back-end traffic wouldn't be duplicated, and if it's a suitably positioned command it could be done after all bogus traffic has been dropped from the inbound query stream, making it less likely to be problematic under abuse conditions. A method for cache insertion is still required at the very core of this concept, and the plumbing around inter-system communication is much more complex, of course. Maybe someone else has already asked for this - haven't dug through the tickets in detail.

In short: if one dnsdist instance in a cluster hears a request and gets an answer (or doesn't) it then sends the cache data to all other participating dnsdist instances by directly inserting the answer into their packet caches. If the sharing system fails for some reason, it has no effect on the operation of each independent dnsdist instance.

shared cache concept: https://github.com/PowerDNS/pdns/issues/12946

[rgacogne]

To be clear I like your idea very much! This is not a very high priority for now so it might take a while, but I would be very interested in merging related PR(s) if someone is motivated :-) One possible way of doing this that would be very flexible would be to provide a Lua hook that is called every time something is inserted into the packet cache. This hook could then use the existing Lua bindings (newNetworkEndpoint and friends, we might need to extend them to support multicast) to send a payload over the network (we might need some bindings for authenticated encryption?), which could then be received by another dnsdist (newNetworkListener and friends) and then inserted into the local cache (requires a new Lua binding to insert into the cache). A different way would still use the packet cache insertion hook and new endpoint of the REST API to insert into the cache. I'm milestoning this to 1.10 since I don't think I will have the time to work on this before 1.9.

[johnhtodd]

I'm thinking that having both a "pull" model as well as a "push" model would be a more complete way to do this, based on your comments above. This also starts to expand this concept quite a bit away from just a simple method to insert objects into the packet cache.

In other words: there should be some way to push data into the packetcache, and that could be for the purpose of pre-populating the cache, or for inserting commonly-used records, or lots of other reasons that I can't envision but probably would be useful. This push method might be able to receive pushed information via a multicast listener, or a unicast socket (with optional encryption and possibly simple shared-key authentication as an obvious add-on.)

But this quickly leads one to consider the "pull" model as well, which is where one dnsdist/recursor instance sends out a notice to associated instances that it has a new request for an unknown query object, and then the other associated instances (if they have the item in their [packet]cache) respond with the results that they know about which would be inserted into the requestor's cache after being transmitted over the network. This could again be unicast (possibly multiple unicast endpoints at the same time) or multicast (first responder wins?) This pushes us towards the concept again of pools having fall-through capability where timers or responses provide no suitable answer from one pool, allowing re-query of the object to the next pool.

[rgacogne]

https://dnsdist.org/advanced/asynchronous-processing.html should provide the building block for the "pull" model once the Lua bindings I mentioned earlier are in place.

[johnhtodd]

In discussions today here internally, the concept of how to load a "warm-ish" cache into dnsdist at start. I came back around to this concept as it is most of what we're talking about, but our ultimate need is both single items as well as bulk load. There is already a way to "dump" the cache; we just need an iterative method to load those objects in at dnsdist launch. We'll probably look at this in the coming months as code to commit, but we might be ham-fisted in the approach. @rgacogne - any suggestions on how we might do this in an acceptable way?

[omoerbeek]

For the Recursor it makes more sense to prime the record cache with data from another Recursor. I'm currently studying/POCing a way to achieve this.

[rgacogne]

I think it would make sense to have a method on a packet cache object to insert a single entry into the packet cache. It has to provide:

• the payload (base64-encoded?)
• the qname
• the qtype
• the flags in the query's header
• the validity (TTD)
• the time the query was inserted (so we can adjust the TTL)
• whether the query was received over TCP or UDP
• optionally the subnet if the entry is ECS-scoped

I would not be opposed to also having a method to load a lot of entries from a file, for bulk loading, but single entry binding is clearly needed.

[johnhtodd]

I agree that the ability to insert a single record is the correct place to start, and the list you provided looks like the basis for what would be required. If the single record method of importation is done with careful consideration, then it should not be difficult to make a method that is fast and iterative as a next feature.

There already exists the ability to dump a cache to file (https://dnsdist.org/reference/config.html?highlight=dump#PacketCache) - is there enough data in each line to use that as an ingestion method? Maybe not quite. Here's an example of what those lines contain:

www.quad9.net. 1157 A ; rcode 0, key 2049581324, length 58, received over UDP 1, added 1728663802 www.quad9.net. 1157 A ; rcode 0, key 561439903, length 58, received over UDP 1, added 1728663764

(one of those is with a "+subnet" addition to the 'dig' command - so two different keys, one which contains ECS and one without)

Notably absent in the dump information is the data to which the key references, which is the vital part of what would be needed in any third-party or built-in tooling that would import records. Am I missing the method to also dump the keys? They do not appear in the file.

[johnhtodd]

One reason we are interested in this (among many!): We often need to restart dnsdist (since it does not support "reload") and this causes the packet cache to be flushed, which in some locations causes significant client slowness or pressure put on back-end resolvers. If we could dump the cache, do the restart, and then re-load the cache, even if some items have expired, this would still be better than starting from zero on a restart. If a better method of restarting dnsdist is developed that doesn't drop the cache, the ability to insert/load items into the cache is still useful in many other contexts - this is not the only reason we have interest.

[rgacogne]

Notably absent in the dump information is the data to which the key references, which is the vital part of what would be needed in any third-party or built-in tooling that would import records. Am I missing the method to also dump the keys? They do not appear in the file.

I'll look into it, and if some keys are indeed missing I'll fix it.

[rgacogne]

We often need to restart dnsdist (since it does not support "reload")

Out of curiosity, which parts of the configuration are you usually altering? I'm wondering if I can add some console command that would help, although I would still very much like to be able to dump/restore the packet cache.

[johnhtodd]

We often need to restart dnsdist (since it does not support "reload")

Out of curiosity, which parts of the configuration are you usually altering? I'm wondering if I can add some console command that would help, although I would still very much like to be able to dump/restore the packet cache.

It's not always the same thing. Yesterday, for instance, I was changing (frequently) destinations for dnstap data. dynblock rules, query counters, and hopefully soon various svcb behaviors. If we built extensive logic into the config that called out to environment variables, I suppose we could avoid some of this, but I think we still need to restart even with env changes. Using a dynamic call of some type seems like it would slow things down significantly.