Expose workload cluster capabilities/attributes

[maleck13]

A cluster that is being added to KCP, may have existing capabilities and inherent attributes. During the life of a synctarget, it may gain new capabilities and possibly attributes. These can provide valuable information to API Providers that need to make decisions based on underlying attributes.

Example Attributes:

• Geo (what geo was this workload placed into)
• Cloud (which cloud was this workload placed onto)

Example Capabilities:

• Some additional workload / operator has been installed on the physical cluster that provides a new capability
• An API/Service provider has installed a workload that provides a capability to a location in response to an API object being created within a workspace. Other workspaces using this location should be able to leverage this capability

Use Case

As an API provider, my service needs to make decisions based on key attributes of a chosen synctarget.

Examples:

• I am provisioning a data base and based on what Geo a workload is synced to will determine where I should create that database instance.
• I am configuring DNS and want to setup Geo based responses, I need to know which Geo(s) the workload has been placed into
• I provide an API that allows the user to weight traffic to go to one cloud provider over another, I need to know which cloud providers a workload has been synced to.

Options

• Expose a set of supported attributes that can be set on the sync target and become available when a workload is placed. These would be a well known set of key values similar to those applied to nodes https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesioregion

Something we experimented with locally was set these attributes directly on the workload keyed by synctarget. This is similar to how the status works. We set these manually on the Ingress object during the experiment experimental.attributes.workload.kcp.dev/5MivhNIs7DjM7dK95I2K7TpWe7aUGMU4WHqjWn: {"geo":{"continent":"EU", "countryCode":"IE"}}

Another option may be to provide a read only view within a virtual workspace to the synctarget(s) a workload has been placed on and provide an API to look that up from a placed resource.

[maleck13]

@sttts I have tried to capture use cases for exposing capabilities cc @ncdc @chirno

[sttts]

Please compare against https://github.com/kcp-dev/kcp/pull/1084, the MVP for location API.

[maleck13]

So on location we have an AvailableSelectorLabels but this looks to be populated by "hand" and not based on capabilities expressed by the synctarget. Would it make sense for a synctarget to support capability labels that can be aggregated to the location? I am thinking about a use case where a controller needs to understand something about the synctarget and a developer needs to understand something about a location. So in the case of GLBC, we are looking at allowing more control over how traffic is weighted and exposing an API around that. So allowing the user to express how they want traffic to be weighted but without knowing the specifics of the individual clusters. One thing that I imagine could be tricky is if a controller needs to read a synctarget to make a decision. I think ideally this would not be required. If a synctarget where labelled with a set of capabilities e.g kcp.dev/capability/cloud=aws would it be reasonable to propagate that information onto any resource being synced to that cluster?

So two questions

1) Should a synctarget be able to inform a location of its attributes/capabilities 2) How should a controller that wants to make decisions based on these attributes get access to them. Should these attributes be reflected onto any synced resource once it has been scheduled (this perhaps all fits under advanced scheduling)

More detail can be found here and use cases:

https://github.com/Kuadrant/kcp-glbc/issues/138 https://github.com/kcp-dev/kcp-glbc/issues/414

[sttts]

Would it make sense for a synctarget to support capability labels that can be aggregated to the location?

This is not the pattern that was behind the API. Instead availableSelectorLabels is meant to be a hand-curated set of label keys and corresponding values. Basically this is an API the compute service provider (owner of the location workspace) has to define.

What we can do is to define conventions and then the syncer supports reporting the decided labels to the SyncTarget object.

Should a synctarget be able to inform a location of its attributes/capabilities

No. A location comes first and stays (it's the user API). SyncTargets come and go. A location is an abstraction of sync targets. A location must be configured in such a way that it supports the properties (AvailableSelectorLabels) that should be exposed for scheduling.

How should a controller that wants to make decisions based on these attributes get access to them. Should these attributes be reflected onto any synced resource once it has been scheduled (this perhaps all fits under advanced scheduling)

Would rather say that the placement could have more detailed info about the chosen synctarget. But we must be careful not to leak information that isn't meant to be public. Also note that a user must note be aware of SyncTargets at all. The location is the concept the user should know and use.

[maleck13]

What we can do is to define conventions and then the syncer supports reporting the decided labels to the SyncTarget object.

Right this makes sense and had thought that might be the way to do it, so when installing the syncer configure it with a set of label values that meet the convention and those get added to the corresponding synctarget

Would rather say that the placement could have more detailed info about the chosen synctarget. But we must be careful not to leak information that isn't meant to be public. Also note that a user must note be aware of SyncTargets at all. The location is the concept the user should know and use.

Yes I think this could work. So once a synctarget is chosen, then any attributes that are set and meet the required convention could be added to that placement object and this would allow a controller that had perms to read the placement to make a decision based on those attributes.

No. A location comes first and stays (it's the user API). SyncTargets come and go. A location is an abstraction of sync targets. A location must be configured in such a way that it supports the properties (AvailableSelectorLabels) that should be exposed for scheduling

This one is interesting to me. If the syncer can add a set of labels to the synctarget, I have to wonder why those wouldn't inform the availableSelectorLabels especially if they have to meet a specified convention. This would seem to help avoid a location that is advertising something but actually no synctargets in that location support that anymore? IE one place to manage it rather than two

[sttts]

Think of availableSelectorLabels as a allow-list of what of SyncTarget's labels to expose to the user.

[maleck13]

Think of availableSelectorLabels as a allow-list of what of SyncTarget's labels to expose to the user.

Not to get bogged down on this one thing, but would a convention not also work as an allow list? E.G kcp/location/attribute always gets added to any location the synctarget is in?

If we did reflect these attributes onto the placement object would it be keyed by cluster similar to how we do the status?

[maleck13]

@davidfestal I have updated this issue to make it more up to date with current thinking and requirements

[mjudeikis]

/transfer-issue contrib-tmc