axolotl-ai-cloud / axolotl

Go ahead and axolotl questions

https://axolotl-ai-cloud.github.io/axolotl/

Apache License 2.0

7.85k stars 864 forks source link

fix: dont change quant storage dtype in case of fsdp #1837

Closed xgal closed 2 months ago

xgal commented 2 months ago

Description

Motivation and Context

Currently jamba storage dtype changes if not prequant in case of fsdp training and it should act as in dsz3

How has this been tested?

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)