Open YWMditto opened 8 months ago
Hi, sorry for the late reply!
In the implementation, we added a small uniform proportion (c=1e-4/7) to each domain when updating the weights. It's simply a smoothing factor, and does not in practice affect results much.
For example, truncating the loss difference to 0 does not seem to be implemented.![image](https://github.com/princeton-nlp/LLM-Shearing/assets/46778265/279a5e9b-9953-416a-941f-9fe71418ccc4)
https://github.com/princeton-nlp/LLM-Shearing/blob/1386c8f69cfb3bf64896959cf3754d2bf87659c7/llmshearing/callbacks/dynamic_loading_callback.py#L34
And, what is the purpose of this line? https://github.com/princeton-nlp/LLM-Shearing/blob/1386c8f69cfb3bf64896959cf3754d2bf87659c7/llmshearing/callbacks/dynamic_loading_callback.py#L41