search

flashinfer-ai / flashinfer

FlashInfer: Kernel Library for LLM Serving
https://flashinfer.ai
Apache License 2.0
1.08k stars 96 forks source link

Support for Volta / Turing architectures #160

Closed tgaddair closed 6 days ago

tgaddair commented 5 months ago

I saw that support for sm75 / sm70 is listed in progress (https://docs.flashinfer.ai/installation.html) but didn't see an issue to track. Is this something planned in the near-term or further out on the roadmap? Thanks!

aliencaocao commented 5 months ago

its tracked here https://github.com/flashinfer-ai/flashinfer/issues/19 but so far no movements in codebase i think

yzh119 commented 5 months ago

@aliencaocao @tgaddair part of the work has been done in #128 , still some work to do to accommodate the small shared memory size of sm75.

Regarding sm70, I have made some local attempts but the performance is not good because I'm using a software simulation of ldmatrix instrinsic. My plan is to write standalone prefill/decode kernels for sm70 because it supports neither async memory copy nor native ldmatrix intrinsic.

They are still on my TODO list, I suppose I can finish sm75 support soon but it will take some effort to debug and performance tuning on sm70, will try my best..

aliencaocao commented 5 months ago

Thanks for the update, looking forward to sm70.

K-Mistele commented 5 months ago

This would be really really great since I would love to be able to use this on my volta devices

balcklive commented 1 month ago

@aliencaocao @tgaddair part of the work has been done in #128 , still some work to do to accommodate the small shared memory size of sm75.

Regarding sm70, I have made some local attempts but the performance is not good because I'm using a software simulation of ldmatrix instrinsic. My plan is to write standalone prefill/decode kernels for sm70 because it supports neither async memory copy nor native ldmatrix intrinsic.

They are still on my TODO list, I suppose I can finish sm75 support soon but it will take some effort to debug and performance tuning on sm70, will try my best..

Can you push the local attemps for sm70? I'm curious to see, and maybe someone can help to finish it.

sparsh35 commented 1 month ago

This would be great.

zhyncs commented 6 days ago

Turing has been supported with https://github.com/flashinfer-ai/flashinfer/pull/449. Volta currently has no supported plans. Welcome to contribute!