A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.
On a simple transformer model, I am observing 10% training speed improvement with Hopper architecture GPUs but not with Ada ones. Baseline is bf16.
I am using huggingface accelerate to handle everything
from accelerate import Accelerator
from accelerate.utils import FP8RecipeKwargs
accelerator = Accelerator(mixed_precision='fp8', kwargs_handlers=[FP8RecipeKwargs(backend='te')])
model = accelerator.prepare(model)
On a simple transformer model, I am observing 10% training speed improvement with Hopper architecture GPUs but not with Ada ones. Baseline is bf16.
I am using huggingface accelerate to handle everything
Is this the best way to use this library?
Should I explicitly use the
te.Linearlayers?