Ensure H100s can be moved around (most likely through ESI)

[joachimweyl]

Motivation

We will need to have our H100s able to shift from OpenStack to OpenShift to BM. Currently our most promising way to do this is ESI. there are some hurtles to get past before we can do this so we are going to track those here and discuss ways to get over them.

Completion Criteria

A plan set in place to ensure H100s will be able to be moved between all of our offerings.

Description

• [ ] Figure out how to get the H100s on the ESI network
• [ ] Add them to the network
• [ ] test ability to move them in and out of OpenShift/Stack BM on the test servers

Discussion

• Working with UMass to see if they still have older 40G IB switch
• the nodes have ethernet for regular stuff (floating IPs, whatever VLAN esi attaches etc.). They also have 200G IB interfaces that are used for inter-node communication. I’m not sure how that will work with ESI but I’m trying to get a basic IB switch to test things out
  • (the existing A100s have these IB interfaces as well, we’re just ignoring them right now)
  • ESI will not work with infiniband as-is
  • If we write a mellanox driver for the IB switch esi should at least be able to segregate interfaces within vlans, and then we can leave it up to the users to do the host-side setup if they want IB (at least for now)
  • The IB switch should functionally have the same control interface as a corresponding ethernet switch, so I was saying that if a user wants to use IB in their GPU nodes ESI could make a VLAN on the IB switch and put the relevant ports on it like a normal ethernet switch, but touch nothing else. This way we don’t really have to adapt ESI for infiniband, at least initially, since it’s just messing with the switch. The host side will stay unconfigured but if we have documentation on it I’m sure users could bring those interfaces up for their use.
• does infiniband use vlans?
  • they’re called PKEYs in the IB world but they accomplish the same thing.
  • We would need to adapt the ansible tasks to match IB control syntax instead of regular vlans/pkey
  • if we can define the available infiniband networks ourselves, we can always hack the playbooks to have some sort of mapping table from "fake vlan" to "pkey"
  • we would need to make sure that the fake vlan was outside the range of user-creatable vlans
• they're also hexadecimal, which is problematic because the segmentation_id field in neutron is an integer
• What is the control interface for IB, so I’m trying to borrow a switch we can use for testing. I haven’t deployed IB before so it’s somewhat new to me

Completion dates

Desired - 2024-07-10 Required - TBD

[naved001]

While partition keys are analogous to VLAN IDs I don't know how the rest of the things work. Like, could someone just setup a subnetmanager on their host and make changes to the IB fabric?

I'd really like to make sure that isolation is guaranteed on the IB networks before trying to do any of the hacky work in ESI.

[hakasapl]

@naved001 I'm not sure, we'll need to test that once we install the borrowed the switch.

We can borrow an unmanaged EDR IB switch from UMass (Mellanox SB7790). Because it is unmanaged we'll need an external subnet manager, which we can just host on an ESI node for testing purposes.

[tzumainn]

In the meantime, here's the parameters of what I think can be done for this in ESI, as well as some considerations:

• This is DEFINITELY a hack. There is no clean solution to supporting Infiniband that I can see
• Infiniband uses partition keys (pkeys) instead of vlans. Pkeys are hexadecimal, while vlans are integers. The problem is that the segmentation_id that neutron uses for vlans is an integer field. There are two ways around this that I can think of:
  • Convert the hexadecimal value into a decimal value, and use that. This only works if the pkey range is entirely disjoint from the vlan range
  • Have some sort of mapping table from "fake vlan" to pkey
  • Assuming similar functionality between Infiniband and ethernet, either solution would allow us to control Infiniband connections at the switch level using a custom ml2 ansible playbook that takes into account the method of converting segmentation_id to pkey value
  • Allowing users to create their own networks in Infiniband is trickier. It would require
  • being able to segregate the infiniband network in neutron's ml2 configuration; may be straightforward, but might have unintended consequences
  • I think that when a new network is created create_vlan is run on all switches. So both the new Infiniband playbooks and the existing playbooks would have to be updated to somehow ignore these commands if intended for the other network type

Just some of my thoughts, I'd be interested in any discussion!