We want to add a method which is called immediately after RoPE (apply_rotary_pos_emb) in Llama's attention module.
Key considerations:
[Con] We're copying verbatim ~60-80 lines of forward code for each attention module (3 for Llama-2).
[Pro] @sashkboos thinks we'll be able to avoid some unnecessary expensive reshaping when applying the hadamards by having access to this part of the code.
nailimixaM
commented
4 months ago
We want to add a method which is called immediately after RoPE (
apply_rotary_pos_emb) in Llama's attention module.Key considerations:
forwardcode for each attention module (3 for Llama-2).