Open tmm1 opened 3 months ago
modeling_mod.apply_rotary_pos_emb = liger_rotary_pos_emb
this is causing loss calculations to be wildly different for some reason
i will investigate further
TypeError: LigerSwiGLUMLP.init() got an unexpected keyword argument 'intermediate_size'
i was able to fix this issue as follows:
modeling_mod.DeepseekV2MLP.forward = LigerSwiGLUMLP.forward
this is causing loss calculations to be wildly different for some reason
the rope method seems to be modified in deepseek v2?
llama:
cos = cos.unsqueeze(unsqueeze_dim)
sin = sin.unsqueeze(unsqueeze_dim)
q_embed = (q * cos) + (rotate_half(q) * sin)
k_embed = (k * cos) + (rotate_half(k) * sin)
return q_embed, k_embed
deepseekv2:
cos = cos[position_ids].unsqueeze(unsqueeze_dim)
sin = sin[position_ids].unsqueeze(unsqueeze_dim)
b, h, s, d = q.shape
q = q.view(b, h, s, d // 2, 2).transpose(4, 3).reshape(b, h, s, d)
b, h, s, d = k.shape
k = k.view(b, h, s, d // 2, 2).transpose(4, 3).reshape(b, h, s, d)
q_embed = (q * cos) + (rotate_half(q) * sin)
k_embed = (k * cos) + (rotate_half(k) * sin)
return q_embed, k_embed
deepseek v2 use MLA(Multi-head Latent Attention) to reduce the kv cache.
Yeah, deepseekv2 one is quite interesting as it used decoupled RoPE.
For the MLA part, since it mainly target on inference case speed up with absorbed low-rank projection matrices into the original linear matrices. Feel free to first try implementing the layers apart from that and can gradually improve with separate prs. Thanks for the interesting feature request and rapid kick off~
π The feature, motivation and pitch
It would be nice to support DeepseekV2 models. Unfortunately the modeling code is not yet accepted into transformers, and requires trust_remote_code=True
I'm monkey-patching myself for now, and wanted to leave some notes that may be helpful when support is added officially down the road.
One initial issue when swapping in swiglu: