Verify fine-tuning working with multi-gpu with half precision

[gkumbhat]

Description

Verify that we are able to fine-tune a model with multi-gpu sharding (fsdp) with bf16.

Discussion

Provide detailed discussion here

Acceptance Criteria

• [ ] Unit tests cover new/changed code
• [ ] Examples build against new/changed code
• [ ] READMEs are updated
• [ ] Type of semantic version change is identified

[chakrn]

Was this test covered by the finetuning testing @olson-ibm did? If not, is it definitely worth doing or not neccessarily?

[chakrn]

Multi GPU support for finetuning is available, so this specific ticket should not be blocked.