[FSDP] [EPIC] Determine how well FSDP works on Intel Gaudi and AMD MI

[RobotSail]

We want to verify that FSDP works in the following scenarios:

• [x] #203
• [x] #204
• [x] #205
• [x] #206
• [x] #207
• [x] #208

[JamesKunstle]

We have validated that FSDP works fine on AMD cards. Haven't gotten anything to run correctly on Gaudi cards.