Open balabala2023 opened 3 months ago
您好,复现的时候发现这类问题,是否对结果有影响?Mistral-7B-Instruct-v0.2 and are newly initialized: ['model.layers.0.self_attn.ultragist_k_proj.weight', 'model.layers.0.self_attn.ultragist_o_proj.weight', 'model.layers.0.self_attn.ultragist_q_proj.weight', 'model.layers.0.self_attn.ultragist_v_proj.weight', 'model.layers.1.self_attn.ultragist_k_proj.weight', 'model.layers.1.self_attn.ultragist_o_proj.weight'....
hi,是最开始预训练加载mistral模型时么?
是的,pt的时候
那是正常的,这些weight需要训练
您好,复现的时候发现这类问题,是否对结果有影响?Mistral-7B-Instruct-v0.2 and are newly initialized: ['model.layers.0.self_attn.ultragist_k_proj.weight', 'model.layers.0.self_attn.ultragist_o_proj.weight', 'model.layers.0.self_attn.ultragist_q_proj.weight', 'model.layers.0.self_attn.ultragist_v_proj.weight', 'model.layers.1.self_attn.ultragist_k_proj.weight', 'model.layers.1.self_attn.ultragist_o_proj.weight'....