为什么baichuan模型中只有decodelayer，没有encodelayer？

[monkeyshichi]

Required prerequisites

• [X] I have read the documentation https://github.com/baichuan-inc/baichuan-7B/blob/HEAD/README.md.
• [X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [X] Consider asking first in a Discussion.

Questions

为什么baichuan模型中只有decodelayer，没有encodelayer？transformer模型架构中不是还有编码层吗？ 这行代码self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)]) 只有decoderLayer class Model(PreTrainedModel): """ Transformer decoder consisting of config.num_hidden_layers layers. Each layer is a [DecoderLayer]

Args:
    config: BaiChuanConfig
"""

def __init__(self, config: BaiChuanConfig):
    super().__init__(config)
    self.padding_idx = config.pad_token_id
    self.vocab_size = config.vocab_size

    self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
    self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)])
    self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    self.gradient_checkpointing = False
    # Initialize weights and apply final processing
    self.post_init()

Checklist

• [X] I have provided all relevant and necessary information above.
• [X] I have chosen a suitable title for this issue.

[ninehills]

因为这是一个纯 Decoder 模型。

现在火的模型基本都是 Decoder-only 模型，包括 GPT、LLaMA、Falcon、Bloom 等等。

[monkeyshichi]

好的，谢谢！