Validate 1D FSDP2 parity with FSDP1

pytorch / torchtitan

A native PyTorch Library for large model training

BSD 3-Clause "New" or "Revised" License

1.57k stars 143 forks source link

Closed gnadathur closed 5 months ago

gnadathur commented 6 months ago

Validate QPS and numerics parity on 8 GPU devGPU and 64 GPU AWS.

awgu commented 6 months ago

On 8 H100 GPUs:

Eager mode parity verified on minipile dataset over 1k steps with both SAC and AC
Compile mode (needs investigation even for FSDP1)
- AC: loss not converging for both FSDP1 and FSDP2
- SAC and no AC: loss converging but not as well as eager mode

awgu commented 5 months ago

Full numerics validation on 8 GPUs is included in https://github.com/pytorch/torchtrain/pull/165.