system prompt not supported on gemma models

[skunkwerk]

I'm running the minimal example with the google/gemma-2-9b-it model.

Running with vLLM's OpenAI server, I get this error:

litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'object': 'error', 'message': 'System role not supported', 'type': 'BadRequestError', 'param': None, 'code': 400}

Is there a way to run DSPy with models that don't support system prompts?

thanks

[arnavsinghvi11]

Hi @skunkwerk , does this discussion from this LiteLLM issue help? Try setting the prefix to openai/ instead of vllm/

[skunkwerk]

thanks. I am actually using the openai prefix, like this:

model = dspy.LM(f"openai/{self.huggingface_model_id}", api_base="http://localhost:8000/v1", api_key=VLLM_OPENAI_SERVER_API_KEY)

that works fine with a model like Llama 3, but with Gemma 2 it returns that error message. Is there any way to tell dspy to not use system prompts?

[okhat]

Hey @skunkwerk ! You'd have to build an adapter.

You can simply build an adapter that calls ChatAdapter and just replaces "system" roles with "user" roles. It could be done in 5-10 lines in total.

See the migration notebook for info about adapters: https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb

See ChatAdapter implementation: https://github.com/stanfordnlp/dspy/blob/main/dspy/adapters/chat_adapter.py

[skunkwerk]

thanks Prof. @okhat! Here's what I'm going to test out:

import dspy

class GemmaChatAdapter(dspy.ChatAdapter):
    def format(self, signature, demos, inputs):
        messages = super().format(signature, demos, inputs)
        if messages and messages[0]["role"] == "system":
            messages[0]["role"] = "user"
        return messages

dspy.configure(adapter=GemmaChatAdapter())

[okhat]

Nice, @skunkwerk ! Lmk if it works!

[skunkwerk]

Yeah, it worked fine! But then ran into this issue, as vLLM only supports 4k context length with gemma :(. So now trying it out with TGI's OpenAI server