search

CCI-MOC / ops-issues

2 stars 0 forks source link

Questions about DPU requirements #239

Closed larsks closed 3 years ago

larsks commented 3 years ago

From @msdisme :+1: what is the form factor of those DPUs (PCIe x1, x4, x8, x16, x32?, v3 or v4?) Thickness of the card too, is it going to take more than 1 slot for cooling? Talking about cooling, any idea of the energy draw?

pjd-nu commented 3 years ago

Whatever it is, it will fit in a 2U Dell - IIRC there are a decent number of R730s in the latest batch of machines from Two Sigma. The Bluefield-2 appears to be fairly low power (i.e. <75W) so we won't need to order external power cables, which are a Dell-specific part. (they attach to some weird connector on the riser)

Without knowing exactly which model it is, I can't tell if it's low-profile (HHHL) or full profile (FHHL). It's vaguely possible that we'll find ourself with no Dell 2U machines set up for full profile cards, in which case we'll have to buy a few minor parts off Ebay or something.

There's a non-zero chance that they won't work on 1U Dell machines. For the 730s, though - if they don't support what's probably the most common 2U rack mount machine on the planet, I don't know who they're planning to sell these cards too.

larsks commented 3 years ago

The details are all in the original purchase issue, #212:

Based on discusions with Nvidia and Red Hat we are planning to purchase 20 MBF2H332A-AEEOT BlueField 2 DPUs. These are HHHL cards (half height, half length) with dual 25Gb ports.

The cards require a PCIe x8 slot.

pjd-nu commented 3 years ago

We were able to fit one of these in a 730XD: https://www.xilinx.com/products/boards-and-kits/alveo/u280.html That's a FHFL 225W 16-lane card - the Bluefield should be a picnic in comparison.

msdisme commented 3 years ago

we think these may also work in the 620s and 630s - once we get the board we will try them out. Related question - are these meant to be connected to the network in which case we need additional switch ports @larsks, @durandom who can help us answer that?

larsks commented 3 years ago

@msdisme I cannot answer those questions, and I don't think @durandom is involved in the DPU work either. Your best bet is probably to pursue this questions via email (I think Andre Beausoleil or Daniel Riek are the folks planning on using this equipment).