How to evaluate APIbank with local model

AlibabaResearch / DAMO-ConvAI

DAMO-ConvAI: The official repository which contains the codebase for Alibaba DAMO Conversational AI.

MIT License

1.18k stars 186 forks source link

Closed oximi123 closed 8 months ago

oximi123 commented 9 months ago

Is there a way to evaluate the benchmark with local model like LLama or Vicuna using FastChat ?

liminghao1630 commented 9 months ago

@oximi123 Refer to evaluator.py and utils.py to implement a Wrapper, and modify the content of evaluator.py for testing.