Open sh0416 opened 1 month ago
Is there any plan to support varlen interface? It seems useful for batch generation and some kv cache prefetch scheme would enhance the performance.
Is there any plan to support varlen interface? It seems useful for batch generation and some kv cache prefetch scheme would enhance the performance.