search

baichuan-inc / Baichuan2

A series of large language models developed by Baichuan Intelligent Technology
https://huggingface.co/baichuan-inc
Apache License 2.0
4.03k stars 289 forks source link

what is z_loss_weight? #350

Closed AICHENaxx closed 6 months ago

AICHENaxx commented 6 months ago

hi, i found this attribute in the forward func of BaichuanForCausalLM as z_loss = self.config.z_loss_weight * softmax_normalizer.mean(). however, there is no definition of this attribute in the config. my question is how to define this attribute?

baichuan-assistant commented 6 months ago

Please refer to this thread: https://github.com/baichuan-inc/Baichuan2/issues/325