search

QwenLM / Qwen

The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.
Apache License 2.0
13.59k stars 1.11k forks source link

About Embedding Layer #1005

Closed OliverHuang1220 closed 8 months ago

OliverHuang1220 commented 8 months ago

在最初的训练基座模型的时候，请问对于language的torch.nn.embedding层是以什么方式初始化的?

jklj077 commented 8 months ago

随机。

OliverHuang1220 commented 8 months ago

感谢你的快速回复，是embeding默认的正太分布初始化的吗