Select ToFQDN prefixes via a special label

[squeed]

This proposes a new label source: dns. This would then tag prefixes with the ToFQDN selector(s) that match this prefix. E.g. 192.0.2.42/32 -> dns:*.cilium.io

Post-merge follow-up items:

• [ ] Consider a flag limiting the number of fqdn: identities in the local allocator
• [ ] Add alerts for excessive local identities
• [x] Chunk ipcache updates.

[squeed]

I just had another idea: identity pre-allocation.

One of the biggest causes of FQDN tail latency is updating Envoy with new identities for a given selector. What if we pre-allocate an identity for every selector? Then we don't need to update Envoy in most cases! (This is because marking an IP as having an existing identity is just an ipcache update, not a policy update). The only time an incremental policy update would be required is if an IP was newly selected by two selectors.

This would be a huge improvement in tail latency.

[gandro]

I just had another idea: identity pre-allocation.

One of the biggest causes of FQDN tail latency is updating Envoy with new identities for a given selector. What if we pre-allocate an identity for every selector? Then we don't need to update Envoy in most cases! (This is because marking an IP as having an existing identity is just an ipcache update, not a policy update). The only time an incremental policy update would be required is if an IP was newly selected by two selectors.

This would be a huge improvement in tail latency.

I thought about per-allocation as well. It really depends how much overlap there is between selectors. If you have a matchPattern: * selector, it will be part of every identity. So if you just pre-allocate an identity for dns:cilium.io, that identity itself will never be used. Instead, all observed IPs will have two labels: dns:cilium.io,dns:*

Edit: And it's not just overlapping selectors, it's also overlapping IPs: The IPCache entry for 1.1.1.1 might have dns:one.one.one.one,dns:* from a lookup to one.one.one.one, but then can also be returned for a lookup to cf-dns-test.cilium.io and thus might also gain a dns:*.cilium.io label in addition. It really depends on how much overlap there is for pre-allocation to actually pay off.

[squeed]

@gandro

I thought about pre-allocation as well. It really depends how much overlap there is between selectors.

Indeed, and we may with wish to be clever when there is a wildcard selector. Pre-allocation, however, doesn't have any relevant downsides (1 identity per FQDN selector, so O(N) = 100), and it optimizes the policy update case when there is no overlap. Which would be a huge win for tail latency; anything we can do to prevent a policymap + envoy update is an improvement.

[squeed]

To all reviewers: this has been updated based on initial prototype implementation. It is now ready for final review.

[squeed]

We've not been able to come up with a circuit breaker, since the scope for identity allocation is already so small. The existing limitation on the maximum number of names per endpoint is probably sufficient.

[squeed]

@joestringer I think this CFP is probably ready to merge :-). If you'd like to take a final pass, feel free.