Open zeyun-zhong opened 1 year ago
Hi, thanks for providing the code.
I have a question regarding the word embedding and lm-head. In your code, both functions shard the same weight. I wonder if they can have separate weights?
word embedding
lm-head
Thanks for your help!
Hi, thanks for providing the code.
I have a question regarding the
word embedding
andlm-head
. In your code, both functions shard the same weight. I wonder if they can have separate weights?Thanks for your help!