search

namespace-Pt / UltraGist

MIT License
15 stars 2 forks source link

Some weights of MistralForCausalLM were not initialized from the model checkpoint #1

Open balabala2023 opened 3 months ago

balabala2023 commented 3 months ago

您好，复现的时候发现这类问题，是否对结果有影响？Mistral-7B-Instruct-v0.2 and are newly initialized: ['model.layers.0.self_attn.ultragist_k_proj.weight', 'model.layers.0.self_attn.ultragist_o_proj.weight', 'model.layers.0.self_attn.ultragist_q_proj.weight', 'model.layers.0.self_attn.ultragist_v_proj.weight', 'model.layers.1.self_attn.ultragist_k_proj.weight', 'model.layers.1.self_attn.ultragist_o_proj.weight'....

namespace-Pt commented 3 months ago

hi，是最开始预训练加载mistral模型时么？

balabala2023 commented 3 months ago

是的，pt的时候

namespace-Pt commented 3 months ago

那是正常的，这些weight需要训练