[Feature request] Support Ollama's raw mode

[alonsosilvaallende]

With open-weights models, you may wish to bypass the templating system and provide a full prompt. In this case, you can use the raw parameter to disable templating. For example, to use the function calling capabilities of Mistral v0.3, the raw prompt is (see Ollama documentation for Mistral): [AVAILABLE_TOOLS] [{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}}, "required": ["location", "format"]}}}][/AVAILABLE_TOOLS][INST] What is the weather like today in San Francisco [/INST] Ollama already supports raw prompts (see Ollama documentation for raw mode). If I understand correctly, the logic behind inspect_ai is with messages (system_message, user_message). Is it possible to also support raw mode?

[jjallaire]

Right now we use the OpenAI compatibility mode (https://ollama.com/blog/openai-compatibility) for interacting with Ollama. Ideally this would be updated to support tool calling and then this would "just work" (with the added benefit of you not having to form raw tool calling prompts for different models). Same will apply if support for multi-modal inputs/outputs is added (e.g. this just got committed a few hours ago! https://github.com/ollama/ollama/commit/e9f7f3602961d2b0beaff27144ec89301c2173ca).

Taking a look at the Ollama repo, it looks like this might actually be in the works? https://github.com/ollama/ollama/pull/5284

[alonsosilvaallende]

Thank you very much for such a detailed response. Indeed, when this PR gets merged it will be very good. However, with raw prompts one can for example force the model to follow a particular tool, for example in the previous example I could add [TOOL_CALLS] [{"name": "get_current_weather", to the raw prompt to force a particular tool to be used (even if there were several available tools). It gives much more control about what behavior the model can have. I wonder if you would accept a PR in that sense and which parts of the code should I modify to add that behavior (generate, viewer)?

[jjallaire]

The OpenAI API will also allow forcing a particular tool (see docs on tool_choice here: https://platform.openai.com/docs/guides/function-calling/function-calling-behavior).

It looks like that PR got merged: https://github.com/ollama/ollama/pull/5284. There is a note in the discussion requesting support for tool_choice. If that's not there I would recommend trying to give them a PR for that.

My orientation here is to try to get everything properly supported in Ollama's OpenAI layer as that will have broad benefits and keep us from having to sort out the details of the various chat templates for tool calling within Inspect. If you want to implement support for raw mode (either for these cases or other ones you dream up) I would suggest creating an ollama-raw model provider and putting it in its own package. Details on doing this are here: https://inspect.ai-safety-institute.org.uk/extensions.html#sec-model-api-extensions