search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.37k stars 1.03k forks source link

8-bit precision training #1788

Open rasbt opened 5 days ago

rasbt commented 5 days ago

Might be worthile to look into 

from torchao.float8 import convert_to_float8_training
convert_to_float8_training(model)

some time

mtasic85 commented 2 days ago

I watched this week this video: https://www.youtube.com/watch?v=Br07GsnnvWc

This performance is expected:

image

About Embedding layer, it should be investigated if they use appropriate class like this one, or we need to take care of it: https://github.com/pytorch/ao/blob/e7b33bc91c831d10249c1222c8b4b667f18f28b7/torchao/quantization/prototype/qat/embedding.py#L27