Closed LeiWang1999 closed 3 weeks ago
Marlin appears to outperform both cutlass and bitBlas within small i4 shapes. Marlin's CUDA kernel design mandates setting the grid size to the number of streaming multiprocessors (such as 108), whereas cutlass and bitblas adjust it according to the tile size (such as 128). which makes around 50% wave waste.
We should dig further.
The project flute provides a cute version of marlin, https://github.com/HanGuo97/flute , I think that's what we should learn and integrate.
Conclusion: block Reduce can enhance the performance and we enable it for small shapes.
PR #63 involves considerable efforts to implement Block reduction with TensorCore, which is expected to improve the performance of continuous batching with tensorcore. However, the performance is still poorer than trtllm batch dequantize kernel, a comprehensive study is required to understand the gap.
code to reproduce: