search

ModelTC / lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
Apache License 2.0
2.42k stars 195 forks source link

Which version is the comparison of the pressure measurement data provided in the project based on? #78

Open dushulin opened 1 year ago

dushulin commented 1 year ago

The comparison data with TGI is based on what TGI version and startup parameters, as well as hardware.

dushulin commented 1 year ago

I loaded llama-7b in lightllm and tgi respectively, and found that the throughput of tgi-0.9.3 will be higher

hiworldwzj commented 1 year ago

@dushulin We tested on TGI==0.8.4, using the A800 80G graphics card. The latest TGI version has made many optimizations. However, for LightLLM performance testing, it requires a warm-up run because the Triton kernel needs to be compiled. When Triton releases version 2.1.0 officially, we will improve the performance of the decode attention operator, and we expect to achieve a performance improvement of 10% to 20% at that time.

hiworldwzj commented 1 year ago

@dushulin LightLLM aims to be the lightest pure Python inference library. In fact, we also have the version that combines TGI with LightLLM, which can achieve higher performance. At the same time, the current scheduling algorithm of TGI cannot handle scenarios with significant differences between long and short sentences very well. In such scenarios, LightLLM performs better in terms of performance. you can build a test set and try it。

dushulin commented 1 year ago

When testing lightllm, a 120s warmup is performed, the data will be better than non-warmup, but still lower than TGI-0.9.3

hiworldwzj commented 1 year ago

@dushulin Can you give more detail about this, We try it.