search

ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs
Apache License 2.0
470 stars 69 forks source link

Does llmperf support measuring local disk models? What's the meaning of framework in llmperf.py line 355? #11

Closed zhangjiawei5911 closed 7 months ago

zhangjiawei5911 commented 7 months ago

I have downloaded llama2-13b-hf to my local disk. I use this command "python llmperf.py -r 20 -m "../models/Llama-2-13b-chat-hf" to measure the performance of llama2-13b. But an error occurred. Traceback (most recent call last): File "llmperf.py", line 419, in endpoint_config["api_base"] = os.environ["ANYSCALE_API_BASE"] File "/opt/conda/lib/python3.8/os.py", line 675, in getitem raise KeyError(key) from None KeyError: 'ANYSCALE_API_BASE' So， please give me some guidance and advise.

zhangjiawei5911 commented 7 months ago

It seems like I chose the default framework,but the default framework doesn't match my model.

rickyyx commented 7 months ago

You will probably need to setup an Open AI compatible model server to process the requests.

The script currently only works as a mere client.

avnishn commented 7 months ago

going to close this for now as ricky has answered the question.