search

vectorch-ai / ScaleLLM

A high-performance inference system for large language models, designed for production environments.
https://docs.vectorch.com/
Apache License 2.0
377 stars 28 forks source link

[minor] use available memory to caculate cache_size by default. #245

Closed liutongxuan closed 3 months ago

guocuimi commented 3 months ago

Thanks looks good to me. two more changes needed.