How to continue pretrain LLM fp8 with hf_causal_lm

mosaicml / llm-foundry

LLM training code for Databricks foundation models

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Apache License 2.0

3.84k stars 503 forks source link

How to continue pretrain LLM fp8 with hf_causal_lm #1261

Open YixinSong-e opened 4 weeks ago

YixinSong-e commented 4 weeks ago

When I continue pretrain HF models with fp8, there is an error: TypeError: ComposerHFCausalLM.init() got an unexpected keyword argument 'fc_type'

dakinggg commented 3 weeks ago

This needs significantly more information to help you. Please provide what you are trying to run, what you modified in the yaml from the examples, and what the output of that is.

That being said, we only have support for fp8 when using ComposerMPTCausalLM at this time.