Closed yzh119 closed 2 months ago
@yzh119 Lol. Your commit titles are hilarious. But might want to censor the output a little. Would hate to have this awesome project be filtered out due to "safety" guidelines by some major corps.
@Qubitium sorry about it, I'll squash all intermediate commits.
These titles are meaningless and I don't want people get bothered by them, and thank you again for your reminder, as an open-source project I should be more professional. I apologize if anyone get notified by any of these commits..
I did a rebase and squash, now random commit messages are cleared, thank you again for your reminder.
I'll wrap up this PR soon, and GQA kernels could be further optimized with these changes.
Currently our warp layout in prefill operators is fixed to 4x1. However, this schedule is sub-optimal when the query length per request is less than 64:
A threadblock-level synchronization function is required to merge states across warps for both cases.
Other changes:
(blockDim.y, blockDim.z)
).