search

PygmalionAI / aphrodite-engine

Large-scale LLM inference engine
https://aphrodite.pygmalion.chat
GNU Affero General Public License v3.0
1.15k stars 127 forks source link

feat: add cuda sampling kernels for top_k and top_p #828

Closed AlpinDale closed 2 days ago

AlpinDale commented 2 days ago

Is faster than Flashinfer's, at least. Still rather disappointing.

Baseline:

Processed prompts: 100%|████████████| 2048/2048 [01:07<00:00, 30.51it/s, est. speed input: 31208.74 toks/s, output: 3904.91 toks/s]
Throughput: 29.51 requests/s, 33996.89 tokens/s

PR:

Processed prompts: 100%|████████████| 2048/2048 [01:07<00:00, 30.20it/s, est. speed input: 30892.24 toks/s, output: 3865.30 toks/s]
Throughput: 29.22 requests/s, 33660.42 tokens/s

Flashinfer:

Processed prompts: 100%|██████| 2048/2048 [01:07<00:00, 30.16it/s, est. speed input: 30853.63 toks/s, output: 3860.47 toks/s]
Throughput: 29.19 requests/s, 33631.21 tokens/s