Use native system instructions for Gemini 1.5 models

[rotemdan]

The current code in google_manifold_pipeline adds a "system"-like prompt by prepending the request with a message containing the system prompt:

if system_message:
    contents.insert(0, {"role": "user", "parts": [{"text": f"System: {system_message}"}]})

On Gemini 1.5 models, Google added support for native system instructions. In the Python API it is passed as an argument for the constructor:

model = genai.GenerativeModel(model_name=model_id, system_instruction=system_message)

The equivalent in the REST API is documented here:

curl https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent?key=$GOOGLE_API_KEY \
-H 'Content-Type: application/json' \
-d '{ "system_instruction": {
    "parts":
      { "text": "You are Neko the cat respond like one"}},
    "contents": {
      "parts": {
        "text": "Good morning! How are you?"}}}'

In my patch I added support for it when the string gemini-1.5 is part of the model id:

            if "gemini-1.5" in model_id:
                model = genai.GenerativeModel(model_name=model_id, system_instruction=system_message)
            else:
                if system_message:
                    contents.insert(0, {"role": "user", "parts": [{"text": f"System: {system_message}"}]})

                model = genai.GenerativeModel(model_name=model_id)

Gemini 1.0 and Gemma models don't support system instructions, so they will fall back to the other approach. Note: after trying to use system_instruction with Gemini 1.0 I got an error from the server, but both Gemma models (gemma2-9b-it and gemma-7b-it) did accept the instruction without error and showed knowledge of it. Officially they don't support system instructions, so it could be a workaround prompt on the server.

Future models, say, like gemini-2.0 will need to be added manually when they become available on the API, otherwise they will fall back to the other approach.

The information returned by the API unfortunately doesn't indicate if the model supports system instructions. Here is for example, the information for Gemini 1.5 pro (returned via https://generativelanguage.googleapis.com/v1beta/models):

   {
      "name": "models/gemini-1.5-pro",
      "version": "001",
      "displayName": "Gemini 1.5 Pro",
      "description": "Mid-size multimodal model that supports up to 2 million tokens",
      "inputTokenLimit": 2097152,
      "outputTokenLimit": 8192,
      "supportedGenerationMethods": [
        "generateContent",
        "countTokens"
      ],
      "temperature": 1,
      "topP": 0.95,
      "topK": 64,
      "maxTemperature": 2
    },

[justinh-rahb]

Excellent, let's get a version number bump to 1.3 and I'll merge this.

[rotemdan]

The two Gemma models do seem to work with the system instruction passed, but I can't really know if all others will.

It's technically possible to add a check for them as well, but I don't know if it matters, since probably most people using the API are using it for the larger models (except for testing maybe).

I guess the script may need to be edited further in the future for those kind of things.

[justinh-rahb]

I think you're right, this should cover probably 90% of most people's usage. Thanks! 🙏

[rotemdan]

Now I realized gemma-7b-it and gemma2-9b-it actually originated from the Groq API, not Gemini. This change doesn't apply to them at all.

I got confused there due to Gemma models appearing on Google AI Studio. The Gemini API currently only provides Gemini models, not any Gemma models.

So, this change does currently apply to 100% of use cases. No need for any other check until a newer version number of Gemini is released.

[justinh-rahb]

I also ported this enhancement to the Functions version: https://openwebui.com/f/justinrahb/google_genai