Use FastChat's prompt templates for huggingface models

[krrishdholakia]

What's the problem? (if there are multiple - list as bullet points) "Prompt templates: One thing you missed: I really don't think the way you are implementing support for HF models is good. From the quick look I had here you just take the messages from the user & assistant and concatenate them in some really simple way (except for llama2-chat where you did implement support for the template for some reason).

That's not how these models are supposed to be used. Different models need a different way to translate the conversation into a prompt string. For an example on how to actually do this correctly, have a look at this file from the popular FastChat repository. I personally don't use their prompt templates and implement them myself, but that has to do with my requirement for evaluation which might not apply for you.

I think for FastChat, you might be able to also just import their library and use their prompt templates. Or copy them over if you want. Because I really wouldn't recommend implementing them all yourself. But just concatenating the message contents like that isn't correct and won't give actual good outputs."

• https://github.com/FastEval/FastEval/pull/81

[krrishdholakia]

FastChat's prompt templates - https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py#L334

[krrishdholakia]

• Focus on support for the TGI models
  • [ ] Bloom
  • [ ] Flan-T5
  • [ ] Llama
  • [ ] OPT
  • [ ] Falcon
  • [ ] MPT
  • [ ] Llama2
  • [ ] Code Llama

[krrishdholakia]

Expose functionality for a user to set their own custom prompt template for huggingface models.

[krrishdholakia]

This is now done - https://docs.litellm.ai/docs/providers/huggingface#models-with-prompt-formatting