search

neuralmagic / guidellm

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
Apache License 2.0
159 stars 11 forks source link

[bug] Model without system prompt fail #52

Closed philschmid closed 1 month ago

philschmid commented 2 months ago

Hello,

When trying to benchmark google/gemma-2-9b-it i ran into an issue. It seems like guidellm uses [system](https://github.com/neuralmagic/guidellm/blob/f7947110ae1446ce765d6019cc4ab0a31d335923/src/guidellm/backend/openai.py#L109) message to sent request, which leads to 

Template error: syntax error: System role not supported (in <string>:1)

I tried to change to user but this leads to pydantic validation errors. Trying to debug this but any pointers or help is appreciated.

markurtz commented 1 month ago

Hey @philschmid, I landed the chang for role to user, are you still seeing an error pop up and if so do you have more details on it?

philschmid commented 1 month ago

Nope I worked from the fork i had and it worked, will confirm from main

philschmid commented 1 month ago

Tested on commit works as well.