dra-evolution: partitioning of devices

[pohly]

This was excluded from https://github.com/kubernetes-sigs/wg-device-management/pull/14 to limit the scope. It's a stretch goal for 1.31.

/assign @klueska @johnbelamaric

[johnbelamaric]

In the 1.31 KEP, we included the APIs defined in https://github.com/kubernetes-sigs/wg-device-management/pull/27, but Mrunal and Tim raised legitimate concerns with the ...verbosity... of that API.

I can think of a few alternatives we can debate, and will propose them as separate PRs in this repo.

cc @thockin @mrunalp @pohly @klueska

[johnbelamaric]

Here are some options. Each of the options 2+ are built on top of option 1.

I suggest looking at the file dra-evolution/testdata/pools-two-nodes-dgxa100.yaml in each PR. This is an example YAML for two 8 GPU servers based on the NVIDIA simulated devices. Real world will be similar, but probably add MORE attributes.

Option 1

• As in the 1.31 KEP
• No attributes shared among devices
• Flattens (de-normalizes) all physical GPU shared resources into a single array of SharedCapacity, using capacity name to differentiate (e.g., gpu-0-memory-block-0)

• 27

• YAML for two DGXA100 nodes (each with eight A100 devices)
  • 341k
  • 12661 lines

Option 2

• Similar to Kevin's original proposal, as well as to this
• No attributes shared among devices
• Shared capacity for each physical GPU are grouped into named sets, rather than embedding the physical GPU name in the capacity name as in Option 1
• Consumption of those per device then name the specific groups from which they consume

• 34

• YAML for two DGXA100 nodes (each with eight A100 devices)
  • 362k
  • 13525 lines

Option 3

• Like option 1, but share common attributes across all devices, overlaying device-specific ones

• 30

• YAML for two DGXA100 nodes (each with eight A100 devices)
  • 205k
  • 7695 lines

Option 4

• Takes option 3 a step further, and defines a common shape for devices, not just common attributes
• In the non-partitioned case, this amounts to common attributes, so it also compacts things a lot in that case

• 31

• YAML for two DGXA100 nodes (each with eight A100 devices)
  • 29k
  • 1121 lines

Option 5

• Takes option 4 a step further, and allows partitions to be algorithmically generated inside the common shape (addresses issue of many partitions, not just many devices, a la Mrunal's question of "100 memory blocks")

• 32

• Complex, just as expressive as enumeration though, because you an still enumerate with this API (very very ugly though)
• Example yaml is hand crafted in this one, rather than generated from mock dgxa100.
• If we do get to 10x the number of memory blocks, and therefore 100s of possible partitions, it could be much much better than explicity enumeration.
• YAML for two DGXA100 nodes (each with eight A100 devices)
  • 19k
  • 780 lines

Option 6

• A slightly different take similar to option 4, but flattens the shape into a list of "partition templates", and then references each template from the explicitly listed set of available partitions in the devices list.

• 35

• YAML for two DGXA100 nodes (each with eight A100 devices)
  • TBD

Option 7

• A merging of optoins 4 and 6 that pushes all the things that are invariant across nodes into their own object and refers to it from the slice. This avoids objects that grow non-linearly as number of partitions per device increases.
• See https://github.com/kubernetes-sigs/wg-device-management/pull/35#issuecomment-2173627898 for an in depth discussion.
• PR TBD
• YAML for two DGXA100 nodes (each with eight A100 devices)
  • TBD

[johnbelamaric]

FYI, I fixed the accidental merge of the wrong PR, and merged Option 1, which matches the KEP (except in the ResourcePool -> ResourceSlice naming).

I also then rebased all the other PRs on top of that. So, it's easier to see the deltas between the KEP and each of the options 1-4.