Add Generic Traffic Ops Config Endpoint

[rob05c]

Currently, the Grove config generator (grovetccfg) uses generic API endpoints, which have far more data than it needs, and are far slower than necessary. A new endpoint in traffic_ops_golang getting only the data needed to configure the cache will be significantly faster, and use significantly less bandwidth.

Generic endpoints are slow and use unnecessary bandwidth, which makes Delivery Service updates more expensive and slower.

This should be part of making Traffic Control more cache-agonstic, and this endpoint should be extended to include everything necessary to configure both ATS and Grove

[ocket8888]

this applies to the ORT/t3c/atstccfg system as well, now.

[rob05c]

I'll also note, this is less of an issue than it used to be. Especially with the transition from DeliveryService-Servers to Topologies.

When I made this Issue, I don't think I or anyone realized literally 99% of the data for both cache config and the CRConfig was DeliveryServiceServers, which is a mathematical Outer Product. With DSS gone, "far more data than it needs" isn't very much, and scalability of TO and network traffic both really aren't issues, even for very large CDNs.

With getting rid of DSS solving the technical issues, there's a lot of advantage in keeping using the standard API endpoints. Cache config (both Grove and ATS) doesn't require a custom endpoint, doesn't require changing TO to support new things or new features, other than those features' own native data.

If we make a "Cache Config Endpoint," I strongly suspect it'll be a constant race, every time we need a new feature in t3c that needs existing data it didn't need before, calling the native Endpoint because t3c is safer, quicker, and more canary-able to deploy than TO, then adding that data to the "Cache Config Endpoint," then changing t3c to use that if it exists and the old if it doesn't, then finally removing the native API call when it's 2 major versions back and all TO versions supported by t3c have the new data. Or we can avoid all that, by just using the native real-data API.

There are advantages to the "Cache Config Endpoint." But with the technical issues resolved by getting rid of DSS, I think there are a lot of advantages to continuing to use the native API. We should weigh the pros and cons of each.

[rawlinp]

Personally, I really like the method that TR uses, which is similar to what this "Generic Traffic Ops Config Endpoint" would be. With few exceptions, TR gets all of its config from a single endpoint which is basically unversioned (so we don't really have to jump through Semantic Versioning Hoops), and if we wanted to, it would be very easy to cache the CRConfig in TO and simply check a timestamp in the DB to know if it has changed with each request. This would offload a tremendous amount of DB load if we were to do this for caches as well, since they make up the vast majority of TO API requests (via t3c). An endpoint like this would also lend itself nicely to server snapshots.

But, like you said, if we were to eliminate DSS in favor of topology-based assignments, that would also alleviate a lot of the DB load.

[zrhoffman]

Related: #4708