search

YujieLu10 / LLMScore

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
121 stars 8 forks source link

Have you tried other public LLMs like Vicuna? #2

Closed ilovecv closed 1 year ago

ilovecv commented 1 year ago

Hi,

I am wondering if you tried other public LLMs like Vicuna except GPT3.5 and GPT4? How about the performance? Thanks?

YujieLu10 commented 1 year ago

In Table 3 in the paper, we report comparisons between GPT3.5 and GPT4. Though GPT4 performs the best in most cases, other open-source LLMs can also obtain decent results. Also, we just released the Vicuna support. Have a try by setting the LLM_ID to GPT-3.5-turbo or vicuna.

linzhiqiu commented 1 year ago

From the code it seems vicuna is not yet supported?

YujieLu10 commented 1 year ago

From the code it seems vicuna is not yet supported?

It's supported, after starting the Vicuna service, you can run llm_score.py by setting LLM_ID as "vicuna". Sorry for the confusion of the comment about "api_key not supported" at https://github.com/YujieLu10/LLMScore/blob/master/llm_evaluator/evaluation_instruction.py#L25 The api_key is not required to use the Vicuna, so we just follow the FastChat repo to set it as "EMPTY".