k8s-ci-robot
commented
2 days ago @dmitsh: The label(s) sig/networking cannot be applied, because the repository doesn't have them.
In response to [this](https://github.com/kubernetes/enhancements/issues/4962):
>### Enhancement Description
>
>- One-line enhancement description (can be used as a release note):
>Standardizing Cluster Network Topology Representation
>
>- Kubernetes Enhancement Proposal:
>- Discussion Link:
>
>- Primary contact (assignee):
>@dmitsh
>@sanjaychatterjee
>
>- Responsible SIGs:
>/sig networking
>
>- Enhancement target (which target equals to which milestone):
> - Alpha release target (x.y): 0.1
> - Beta release target (x.y):
> - Stable release target (x.y):
>- [ ] Alpha
> - [ ] KEP (`k/enhancements`) update PR(s):
> - [ ] Code (`k/k`) update PR(s):
> - [ ] Docs (`k/website`) update PR(s):
>
>
>
>_Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently._
>
>## Summary
>
>This document proposes a standard for declaring cluster network topology in Kubernetes, representing the hierarchy of nodes, switches, and interconnects. In this context, a switch can refer to a physical network device or a collection of such devices with close proximity and functionality.
>
>## Motivation
>
>Understanding the cluster network topology is essential for optimizing the placement of workloads that require intensive inter-node communication. Currently, there is no standardized way to represent this information in Kubernetes, making it challenging to develop control plane components and applications that can leverage topology awareness.
>
>This information might be useful for various components and features, including:
>- Pod affinity sections in deployment and pod specs
>- Kueue network-aware scheduling
>- Future development of native scheduler plugins for topology-aware scheduling, for example
> - topology aware gang scheduling plugin
> - DRA scheduler plugin
>
>## Cluster Topology Sources
>
>Cluster topology information can be derived from various sources:
>- Provided directly by a Cloud Service Provider (CSP)
>- Extracted from a CSP using specialized tools like "topograph"
>- Manually set up by cluster administrators
>- A combination of the above methods to ensure comprehensive coverage
>
>## Proposal
>
>We propose new node label and annotation types to capture network topology information:
>- Network Topology Label
>- Network QoS Annotation
>
>### Network Topology Label
>Format: `network.topology.kubernetes.io/: `
>- ``: Logical type of the network switch (can be one of the reserved names or a custom name)
> - Reserved names: `accelerator`, `block`, `datacenter`, `zone`
>- ``: Unique identifier for the switch
>
>### Network QoS Annotation
>Format: `network.qos.kubernetes.io/switches: `
>- ``: A JSON object where each key is a switch name (matching the network topology label) with a value containing:
> - `distance`: Numerical value representing the distance in hops from the node to the switch, required
> - `latency`: Link latency (e.g., 200 ms), optional
> - `bandwidth`: Link bandwidth (e.g., 100 Gbps), optional
>
>This structure can be easily extended with additional network QoS metrics in the future.
>
>## Reserved Network Types
>We have introduced reserved network types to better accommodate common network hierarchies. These reserved network types include the following predefined names and characteristics:
>
>1. `accelerator`: Network interconnect for direct accelerator communication (e.g., Multi-node NVLink interconnect between NVIDIA GPUs)
>2. `block`: Rack-level switches connecting hosts in one or more racks as a block.
>3. `datacenter`: Spine-level switches connecting multiple blocks inside a datacenter.
>4. `zone`: Zonal switches connecting multiple datacenters inside an availability zone.
>
>When using reserved network types, Network QoS Annotations become optional. In the absence of these annotations, it is assumed that performance within each network layer is uniform.
>
>The scheduler will prioritize switches according to the order outlined above, providing a standardized approach for network-aware scheduling across a range of configurations.
>
>If provided, Network QoS Annotations can be used to refine and enhance the details of link performance, enabling more precise scheduling decisions.
>
>### Example of Network Topology Labels with reserved network types:
>
>```
>network.topology.kubernetes.io/accelerator: nvl72-a
>network.topology.kubernetes.io/block: block-b
>network.topology.kubernetes.io/datacenter: dc-c
>network.topology.kubernetes.io/zone: zone-d
>```
>
>### Example of Network QoS Annotations that complements the example above:
>```
>network.qos.kubernetes.io/switches: {
> "nvl72-a": {
> "latency": "2us",
> "bandwidth": "100Gbps"
> },
> "block-b": {
> "latency": "50us",
> "bandwidth": "40Gbps"
> },
> "dc-c": {
> "latency": "500us",
> "bandwidth": "20Gbps"
> },
> "zone-d": {
> "latency": "1ms",
> "bandwidth": "10Gbps"
> }
>}
>```
>
>## Extensibility and Future-Proofing
>
>This proposal is designed with extensibility in mind, enabling the use of custom network types. This ensures that the standard can adapt to future advancements in cluster networking without requiring significant overhauls.
>
>For custom network types, Network QoS Annotations are required, with distance being the minimum mandatory metric. Specifying latency and bandwidth is optional, but including them can offer a more detailed view of link performance, enabling more efficient scheduling decisions.
>
>### Example of network topology with custom network types
>
>#### Node Labels:
>```
>network.topology.kubernetes.io/area: sw-a
>network.topology.kubernetes.io/sector: block-b
>network.topology.kubernetes.io/center: center-c
>```
>
>#### Node Annotations:
>```
>network.qos.kubernetes.io/switches: {
> "sw-a": {
> "distance": 1,
> "latency": "100ns",
> "bandwidth": "40Gbps"
> },
> "block-b": {
> "distance": 2,
> "latency": "500ns",
> "bandwidth": "20Gbps"
> },
> "center-c": {
> "distance": 3,
> "latency": "1ms",
> "bandwidth": "10Gbps"
> }
>}
>```
>
Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
adrianmoisey
dmitsh
alaypatel07
ritazh
kannon92
Enhancement Description
One-line enhancement description (can be used as a release note): Standardizing Cluster Network Topology Representation
Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/pull/4965
Discussion Link:
Primary contact (assignee): @dmitsh @sanjaychatterjee
Responsible SIGs: /sig network
Enhancement target (which target equals to which milestone):
[ ] Alpha
k/enhancements) update PR(s):k/k) update PR(s):k/website) update PR(s):