Closed pujiang2018 closed 4 months ago
Refactor resize function in DecoderContext to support Continuous Batching, and removed qkScores member (since it is rarely used and the attention impl. most likely would like to mange the buffer by itself).
Refactor resize function in DecoderContext to support Continuous Batching, and removed qkScores member (since it is rarely used and the attention impl. most likely would like to mange the buffer by itself).