Closed ilovecv closed 1 year ago
In Table 3 in the paper, we report comparisons between GPT3.5 and GPT4. Though GPT4 performs the best in most cases, other open-source LLMs can also obtain decent results. Also, we just released the Vicuna support. Have a try by setting the LLM_ID to GPT-3.5-turbo or vicuna.
From the code it seems vicuna is not yet supported?
From the code it seems vicuna is not yet supported?
It's supported, after starting the Vicuna service, you can run llm_score.py by setting LLM_ID as "vicuna". Sorry for the confusion of the comment about "api_key not supported" at https://github.com/YujieLu10/LLMScore/blob/master/llm_evaluator/evaluation_instruction.py#L25 The api_key is not required to use the Vicuna, so we just follow the FastChat repo to set it as "EMPTY".
Hi,
I am wondering if you tried other public LLMs like Vicuna except GPT3.5 and GPT4? How about the performance? Thanks?