search

huggingface / nanotron

Minimalistic large language model 3D-parallelism training
Apache License 2.0
1.14k stars 107 forks source link

[FP8 Training] A single forward and backward pass for a linear in FP8 #56

Closed xrsrke closed 7 months ago

xrsrke commented 7 months ago

This PR contains a single FP8 Linear layer supporting both forward pass and backward pass. The next PRs will enable FP8 end-to-end training.

image