Move Ceph cluster from PRB to MGHPCC and promote it into new research Ceph cluster

[msdisme]

@pjd-nu and @naved001 discussed over lunch on Jun 16 - Peter this ticket is to capture your thoughts and discuss what support or space may be needed.

this is what Erik set aside for this 5 FX2s 10 FC630s 8 FD332s

[joachimweyl]

Current plan:

• 3 FX2s (2 FC630s & 2 FD332s in each) (6 each all together)
  • 8 SSD & 8 HDD in each FD332 (16 each in each FX2) (48 each all together)
    • What hard drives do they currently have in them?
• 1 FX2 (4 FC630s)
• 1 FX2 (spare)
• 2 FD332s (spare)

Needs to get done for plan:

• [ ] Get Flax to bring back some hardware from the 2 Sigma donation
  • 5 FX2s
  • 10 FC630s
  • 8 FD332s
• [ ] Gather Hardware from old PRB cluster
  • 32 SSD
  • 48 HDD
  • 40 GB Switch
  • PCIe-based NVMe drives
• [ ] Order extra hardware
  • 16 SSDs
  • all required Sleds (96? do we have any sleds?)
• [ ] Setup Ceph cluster

@pjd-nu "If we wanted to later we could swap 2 of the FC630s to bump it to 8 OSD machines, and either dedicate other machines to monitors or just run monitors on the OSD nodes. I’ve got a 40G switch and 40G NICs for the machines, and would be moving over PCIe-based NVMe drives as well. We would need 1G connectivity for IPMI, and we’d probably want to run a standard 1G network to all the machines as well. (This would let us use my oddball 40G switch without needing to configure it or anything - it would just run an isolated cluster network) I’m assuming you wouldn’t want to use my current 1G switch, which is something dreadful I bought off the shelf at Microcenter. The plan for the cluster is that the grad students would run it as a Ceph cluster the same way we’re running the PRB cluster - periodically blowing it away and re-installing Ceph whenever we want to test against a different version, etc. Any coordination of this would be between grad students."

[joachimweyl]

@pjd-nu is this still the desired plan? we will need to let Flax know soon before they finalize the processing of these units.

[joachimweyl]

@er1p agreed here to set aside:

• 5 FX2s
• 10 FC630s
• 8 FD332s

[pjd-nu]

Just checking on how this is doing, as we'll have to coordinate taking down the PRB cluster and swapping some cards and disks in order to set these up. (note FC630 and FC830 are exchangeable here - if there are differences in the blades, using the lower-performance ones here would make sense)

[joachimweyl]

Harlan McCanne from Flax confirms they have reserved the hardware and will bring it to MGHPCC when we are ready to start work on this project.

[joachimweyl]

@pjd-nu the hardware for this is at Flax. We can ask them to bring it back to MGHPCC when we are ready for it to be used. They wanted to be clear that the FD332s are currently diskless. I believe you already knew this based on your suggestion of gathering disks but it is better safe than sorry.

[pjd-nu]

Still waiting for the eval hardware. Any update???

[msdisme]

Drives @er1p held onto at flax: https://github.com/CCI-MOC/ops-issues/issues/1038#issuecomment-1793843794

[joachimweyl]

we should be able to get all the drives we need from this batch at Flax. We just need to let @er1p know how many we want.

[joachimweyl]

Erik already delivered a test model of one of the FX2s to @pjd-nu 1 FX2

• 3 FC630
  • 2 1.6TB SAS SSD
• 1 FD332
  • 16 1.6TB SAS SSD

[joachimweyl]

not going to start work on creating the cluster until Jan 16 at the earliest. But any of the prep work we can do beforehand is good.

[joachimweyl]

This is the hardware we still need: 3 FX2 7 FC630

• 2 SAS SSDs in front bays each

5 FD332

• 32 1.6TB SAS drives (loose or in FD332)

[msdisme]

Row 4, Pad A, Cabinet 4 is being proposed - @pjd-nu ok?

[joachimweyl]

@er1p can we have the rest of the parts for this cluster sent to the MGHPCC?

[joachimweyl]

@er1p "those servers were delivered yesterday FC630 / storage servers in R4-PA-C04

we over-delivered a small amount four chassis FX2 - request was three eight servers FC630 - request was seven six JBODs FD332 - request was five fifty-six drives 1.6TB SAS/SSD - request was forty-six when we were putting the nodes together, moving around the JBOD units seemed to move the PCI structure around, and seemed to get confusing very quickly, so we figured it made more sense to be consistent - specifically, we created :: compute/FC630 node hosting for up to four FC630 nodes, no JBOD 1 or 2 or 3 or 4 servers - one or two drives each storage/FC630 node hosting for up to two FC630 nodes in the top row hosting for up to two FD332 JBODs in the bottom row 1 or 2 servers - one or two drives each 1 or 2 JBODs - up to 16x drives each so then it was logical to deliver four chassis one compute - two servers FC630 three storage - six servers FC630, six JBOD FD332 56x drives total - sixteen drives (2 per server) and forty drives (16 + 16 + 2 + 2 + 2 +2 in jbod) this way every jbod has at least two drives, so fully tested / working this means that Peter can move the drives around at will - and it means that if he wants to merge the systems that he has on campus, he can just move the servers and loose drives around directly into the rack, he doesn't have to mount another chassis"

[joachimweyl]

@pjd-nu ^

[pjd-nu]

Thanks!