search

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving
https://flashinfer.ai
Apache License 2.0
1.32k stars 121 forks source link

[Bug] data_type default overwrites q_data_type #543

Open cyang49 opened 21 hours ago

cyang49 commented 21 hours ago

https://github.com/flashinfer-ai/flashinfer/blob/78e26e47b95bea994ad2a47e1b1f42810363429c/python/flashinfer/decode.py#L587

Current logic sets default to data_type and the comment suggests that users should set q_data_type and kv_data_type instead.

However, if a user doesn't explicitly set data_type=None, it causes the explicitly set q_data_type and kv_data_type to be overwritten by default data_type=float16. I don't think this is the intended behavior and it's causing confusion.

yzh119 commented 16 hours ago

@cyang49 thank you for spotting this bug!