[RFC][Snoosweek] Support for DTab overrides

[folone]

📚 Background reading

• Linkerd doc
  • Namers
• Finagle doc

💸 TL;DR

This PR demonstrates a simple support for system-wide request-level target override for services. I've tried my best to demonstrate how it would work in the following diagram:

📜 Details

This is a demonstration PR, intended to elicit comments/opinions, not intended for merging as is.

This change does not exactly implement DTabs as described in the docs above – those require some internal knowledge of service/name resolver on behalf of the client constructor. Rather, this change is inspired by those documents. Which means the name "DTab" might not actually be very relevant here. Should this be called something like "ServiceAddressOverrides" instead?

💡 The general idea is as follows:

• We add a new request header that describes the desired overrides. In this implementation, the header (X-DTabs) takes the following form: service.a.prod=>service.a.staging; service.b.prod=>service.b.staging;. In this case we instruct the client to replace all the calls to service.a.prod with service.a.staging and all the calls to service.b.prod with service.b.staging. See the newly added tests for more context.
• The header gets propagated in the contexts, the same way we propagate other headers (such as the tracing headers).
• Whenever we encounter calls to service.a.prod or service.b.prod on our request path, and the above DTab is set, we reroute the requests to service.a.staging/service.b.staging correspondingly.

🧑‍🔬 User stories

• I'm a Product Manager, wanting to test out a feature that one of the engineers on my team is working on. Standing up a staged environment of everything is fine I guess. But I'd prefer that the engineer could stage just the services they are changing, and give me some headers I could plug into ModHeader, so I could use the actual product we're developing in its production environment (except for that one service).
• I'm an engineer building a Snoosweek feature. A bunch of the code I've written is in other teams' codebases, and lacks testing coverage (since it's a prototype, and I only had half a week to tie it all together). Reasonably enough, that other team won't let me merge/deploy the code to be able to demo it, even if I promise to hide everything behind an experiment and roll everything back next week.
• I'm an on-call engineer from a Core Services team during Snoosweek. Dozens of engineers have amazing ideas that involve code changes in my team's codebases. I'm happy to facilitate them demoing those changes, but a lot of these changes have merge conflicts with each other, and I don't exactly want all of those prototypes in our codebase during a code freeze. Wouldn't it be great if they could just stage their own versions of the service?

🔨 Improvements that can/should be made here

• [x] Creating new clients for dtabbed requests is much more expensive than just fetching a client from the pool. We could add an LRU cache for newly created client pools, to avoid re-creating them upon each call.
• [ ] DTabs should be passed around in a parsed form, rather than the string representation I have here, since this requires each successive service to re-parse the header.
• [ ] This PR only implements overriding target addresses for thrift clients. Complete implementation would involve supporting these overrides for all the clients baseplate has (e.g. gRPC).
• [ ] These headers should likely be filtered out from external (outside of Reddit) requests, to avoid leaking existing staged internal service addresses.
• [ ] Tests asserting new clients with address overrides get created when DTab headers are present.

✅ Checks

• [ ] CI tests (if present) are passing
• [ ] Adheres to code style for repo
• [ ] Contributor License Agreement (CLA) completed if not a Reddit employee

[folone]

@kylelemons @konradreiche @pacejackson Thanks for the feedback folks! Looks like there might be some tentative interest in something like this. As Kyle suggested, I'll write up a design doc and try to get a slot in your office hours to discuss how feasible this is within Reddit.

[fishy]

@folone This part won't really work in Reddit's environment:

But I'd prefer that the engineer could stage just the services they are changing, and give me some headers I could plug into ModHeader, so I could use the actual product we're developing in its production environment (except for that one service)

Because it essentially requires traffic between production and staging clusters, which is not allowed.

Basically at Reddit stagings can only talk to stagings and productions can only talk to productions.