Closed LiuXiaoxuanPKU closed 5 months ago
In the example above, if I deliberately make prefix length '0'
prefix_lens = [random.randint(0, 0) for _ in range(batch_size)]
and I find that the output(output
) of flashInfer are all 0s, this is not expected because the append kernel should degrade to pure prefill kernel in that case, and running prefill should output non-zero values.
Fixed! I did not pass the parameters correctly, the output is similar to xformers.
Thanks for the great work!
I'm trying to compare the results of flashinfer BatchPrefillWithPagedKVCache's output with xformers, but got different outputs, this is the minimum reproducible code. I might also misunderstand flashinfer's API, please correct me if I'm wrong, thanks!