search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
185 stars 42 forks source link

Add support for system prompt and messages API through ModelRunner.predict() #205

Closed gilinachum closed 6 months ago

gilinachum commented 6 months ago

System prompt is required for Claude Messaging API and for GPT. FMEval API current predict API doesn't support:

  1. passing a system prompt a long side the user prompt.
  2. Passing messages to support chat prompting and multi-modal prompts.
    The accuracy of these models depends on being able to separate the system and the user prompt.

Current API:

class ModelRunner(ABC):
...
def predict(self, prompt: str) -> Tuple[Optional[str], Optional[float]]:
szl0144 commented 6 months ago

Hi @gilinachum , I met the same issue today, Claude 3 on Bedrock is a Multimodal and the prompt format is different from the text model. I tried to run the benchmark on Claude 3 but failed... If anyone can give me some help?

danielezhu commented 6 months ago

Hi, this issue will be solved with multi-variable prompt templates in the future. As a workaround right now, you can embed the system prompts in the content_template if they are static for each inference call.

gilinachum commented 6 months ago

I see it here. Thx.