search

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving
https://flashinfer.ai
Apache License 2.0
1.1k stars 98 forks source link

QUESTION: C++ API support Ragged Tensor now? #189

Closed yz-tang closed 5 months ago

yz-tang commented 5 months ago

In the c++ unit test, I only found case about paged tensor。 How to use ragged tensor in c++？ When I try use BatchPrefillWithRaggedKVCacheWrapper, it will link error。

yzh119 commented 5 months ago

Yes we support ragged tensors in C++ apis, we add some examples in #190.

Regarding your link issue, you can check whether you have instantiated the templates. We decouple the declaration and implementations to accelerate compilation speed, if you only include the declaration header, then you have to compile the template instantiations and link them properly (see our CMakeLists).