Open lkc1997 opened 5 months ago
I found that during shared-prefix calculation, this kenerl won't use _qoindptr to split batch queries which may cause rope error.
Hi @lkc1997 , we have another few arguments (rope_position
to specify the rope position of each query) in our C++ APIs but we have ported them into PyTorch APIs, I'll do that in next release.
You can process rope outside attention and use attention kernel without rope at the moment.