search

GreenBitAI / green-bit-llm

A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
https://huggingface.co/blog/NicoNico/green-bit-llm
Apache License 2.0
61 stars 7 forks source link

About the DiodeMix optimizer #19

Closed hanhanpp closed 2 days ago

hanhanpp commented 2 weeks ago

The DiodeMix provide different update strategies for different bit parameters. But I find that only 1-bit parameter's update method is spcified, others, e.g. 8-bit parameter, are same as Adam optimzer. Is that right?

yanghaojin commented 2 weeks ago

Correct, the gradient update mechanism is modified based on the AdamW optimizer, and supports the alternating update of quantization-related parameters such as zeros and scales and weight. The update for scales will be released in the upcoming version.