Open amitagh opened 4 months ago
code mistake. not an issue.
This is an issue still there. earlier i thought this is an issue with my script but it is not. axolotl is saving in FP32 at the end of pretraining with lora. Both merged model and adaptor is stored in FP32 precision though base model is in FP16.
Please check that this issue hasn't been reported before.
Expected Behavior
After merge the final merged model should have been of same precision as the base model but it is FP32 this leads to size of the merged model almost doubling as compared to size of the base model.
Current behaviour
After merge the final merged model should have been of same precision as the base model but it is FP32 this leads to size of the merged model almost doubling as compared to size of the base model.
Steps to reproduce
Generate a lora adaptor.
Below is the config i used for Gemma 2B Pretraining with Lora. after pt i try to merge the adaptor generated with base model. In merged models config.json i see torch.dtype as FP32. Thus the size of merged model is 10 GB when base model is of 5GB
Config yaml
Possible solution
No response
Which Operating Systems are you using?
Python Version
3.10
axolotl branch-commit
latest
Acknowledgements