Open tcdowney opened 4 years ago
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/174718773
The labels on this github issue will be updated when the story is started.
@tcdowney I thought Route
CR is a part of Cloud Foundry and I don't understand why it should be less CloundFoundry-specific.
Hello @tcdowney thanks for such a beautifully written issue :tada:. Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf" from their components (e.g cf-api-server, cf-api-worker, etc)?
However, I agree about changing app
objects to what you've suggested as you mentioned we are not using it for app matching.
@tcdowney I thought Route CR is a part of Cloud Foundry and I don't understand why it should be less CloundFoundry-specific.
This is definitely a fair point. There's some value in keeping them similar. It makes the code a little easier to reason about and it provides a familiar experience to folks who have been operating Cloud Foundry for a while. But I think the flexibility provided by generalizing the CRD outweighs these benefits.
The close coupling between the Route CR and the CF v3 Route data model limit what consumers are able to do with the CRD. That's fine for the CF API today since it matches perfectly with its database schema, but for future iterations of CF or for folks that are just experimenting with the Route
CRD it is pretty unfriendly.
What if I'd like to use some other control plane instead of the CF API (maybe something lighter-weight)? I will have to know what destinations[].app.guid
and destinations[].app.process.type
refers to and know that I need to just make stuff up here. Similarly for destinations[].guid
the onus is on me to come up with a unique identifier -- and why does it need to be a guid? Things like that just make it a bit strange of an interface.
By making the Route CRD more generic in these aspects it should result in a more flexible extension point.
Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf" from their components (e.g cf-api-server, cf-api-worker, etc)?
No plans to, and I don't think those should be renamed. This isn't a desire to expunge "Cloud Foundry" specifically, but rather to identify extension points that don't have to be Cloud Foundry-specific. The CF API (and friends) enable the traditional CF UX and I think they have to be pretty coupled to CF out of necessity.
The Route CRD though is an inner abstraction that could enable other workflows alongside CF and this proposal would help that.
Apart from all that, I think it's cool how some of the Contour folks have been able to use their HTTPProxy
CRD to experiment with new ingress features and propose them to the upstream Ingress
v2 / next generation K8s Services APIs work. The Route CRD could be an opportunity to help bring Cloud Foundry concepts/features to the broader Kubernetes community. Generalizing the CRD by removing a few CF-isms will make it easier to understand and digest.
Route CR is a part of Cloud Foundry
remove "cf"
a desire to expunge "Cloud Foundry"
I hope I'm not derailing, but I think I'm detecting an opportunity for better nomenclature here. In this thread people are using "CF" to refer both to our broad organization (the thing the Route CR is a part of) AND the "CF API" (the thing that the Route CR ought nought to be specific to. Sometimes it's used twice in the same sentence but could be read to refer to 2 different things. If we draw the distinction between these 2 things, I think maybe we'd have an easier time communicating about them.
Out of curiosity, does the Cloud Controller (CAPI) team have any plans to remove "cf-[API]" from their components (e.g cf-api-server, cf-api-worker, etc)?
identify extension points that don't have to be Cloud Foundry [API]-specific
coupled to [the] CF [API] out of necessity.
I think the analogous thing here is that there's a tiny bit of code starting to form in cf-api-controllers
that we might decouple from the API. Most of that component is coupled tightly to the CF API, like Tim says, but there's a TINY bit of glue that watches for kpack stack updates and applies their resulting image changes to Eirini's StatefulSets - an operation that could be useful 100% independent of the involvement of the CF API.
One day we may want to extract that thing to be part of an cf-app-controllers
that we might deploy without a CF API or the rest of cf-api-controllers.
Tim's suggested changes to Route
echo this intent: in the Route CRD, what could we decouple from the specifics of the CF API?
First off, thanks for opening this, I really like this direction.
We haven't implemented TCP Routing in cf-for-k8s yet, but when we do, the ability to ask for "random port" would be necessary, and a meaningful enhancement atop anything I see in the rest of the kubernetes ecosystem.
there's good tenancy stuff (e.g. route ownership based on domains, first-come-first serve deconfliction of route names) that is well done in CAPI but could be extended into k8s, if we had something like CF Space being 1:1 with K8s Namespace.
tl;dr
Remove/rename some of the fields in the Route CRD to be less Cloud Foundry-specific.
Background
I want to use this Issue to share some thoughts a few of us have had about the Route CRD and to provide a place for discussion. I don't expect anything concrete to come out of this Issue specifically, but wanted to get some of these thoughts written down somewhere -- especially as y'all begin exploring alternative Ingress/mesh implementations which might require changes to the Route CRD.
The Route CRD (sample
Route
CR) has several fields that are specific to Cloud Foundry -- especially theapp
portion of adestination
:The
spec.destinations[].selector
field is what is ultimately used for configuring routing. Thespec.destinations[].app
field is used for providing a few helpful labels on the createdServices
and for setting headers that are used for app access logging.This design has been useful since it closely mirrors the CF v3 Route API. Unfortunately, this design also tightly couples the
Route
with Cloud Foundry apps and makes it difficult to adapt to other (theoretical) types of routable workloads.In addition to making it a more flexible interface, this has the added benefit of making the
Route
CR serve as an example for enhancements we might want to propose upstream.For example, one day we may want to standardize on some of the next generation Kubernetes Service APIs -- perhaps even use the
HTTPRoute
/TCPRoute
resources directly1 -- and the CFRoute
CR could be something we use to drive out enhancements to that.Or maybe this helps it becomes a more general
ApplicationRoute
that could be used in other PaaS-like contexts.1 - The
HTTPRoute
/TCPRoute
may be too low-level and not something a component like the CF API would want to make directly. I'm mostly using these resources as a straw dog example for a standard we could work toward.Proposal
App Guid / Process Type
Consider removing the required
spec.destinations[].app
attribute and replacing it with several optional attributes that more clearly describe the intent of the field.spec.destinations[].logging_id
- OptionalString
identifier to use for aggregating logs associated with this route. In Cloud Foundry this would be used to tie access logs for a given route to the app they belong to. For Istio this would ultimately end up being a header that gets added to the request.spec.destinations[].headers
- Optionalmap[string]string
containing arbitrary headers to add/modify on the request.Destination Guid
The
spec.destinations[].guid
attribute is a little strange. It is currently being relied on to give the childServices
a unique name. A guid is suitable for this purpose, but it's a little strange that we are relying on the creator of theRoute
to know this and provide the guid. Perhaps we could rename this field toname
(and in Cloud Foundry it's value could remain a guid) and the actual underlying childServices
could be named<route-name>-<random-identifier>
. Essentially the same as how aDeployment
namedcf-api-server
would create aReplicaSet
named something likecf-api-server-c7d8b8f64
.spec.destinations[].name
-String
containing the name of the destinationDomain
The
spec.domain
attribute is a little interesting. I'm not sure what theinternal
route UX will end up looking like, but this might be a field we could revisit as well. That said, it might be generic enough as is.Metadata for Child Resources
The labels we were adding to the child
Service
created by each destination were more of a nice-to-have, but maybe there can be some form oftemplate.metadata.labels
/template.metadata.annotations
that could propagate to child resources. I'm not going to include this on the proposed example since it's not strictly necessary, but still something to think about.Example Route
With these changes a Route CR would look like:
Just wanted to start an open discussion here to throw some of these ideas out there. If this is something y'all might actually do I'd be happy to collaborate on the design via Google Doc, PR, or other means.
cc @zrob who has also 🤔 about this and experimented with the
Route
CR