Identify where traffic is coming from

[xumengpanda]

When a cluster is near overloaded, operators want to understand where traffic is coming from in order to decide how to reduce the traffic.

Two questions to answer:

• [ ] 1) Is the traffic from internal or external workload? For example, are extra reads coming from data distribution's load balancing or clients? If traffic is from the internal workload, operators can temporarily disable some non-must component.
• [ ] 2) When some weird traffic pattern occurs, can FDB tell clients (e.g., RecLayer) which service is likely causing the new traffic pattern? This can help layers team (i.e., FDB clients) to identify the problem quickly.

To answer these questions, we may tag transactions inexpensively and aggregate metrics by tag. The number of tags should be small because the number of services on a cluster is limited.

Once FDB has the aggregated metrics, it also needs a better way to expose those aggregated metrics, which will be solved in Issue https://github.com/apple/foundationdb/issues/2156.

[This idea was proposed and discussed in FDB Offset 2019.]

[dongxinEric]

This might also rely on some changes at the client side to better tag different sources.

[ajbeamon]

This is fairly similar to #1692

[xumengpanda]

I think #1692 is similar to the item 1 in this issue. The item 2 is more finer-granularity tracking, IMO.

I won't mind closing one of them.

[ajbeamon]

Yeah, interestingly #1692 also suggests finer-granularity tracking, but only internally. I think it's the same idea, though, so we could probably safely fold these together.