Design new LLM database schema

simonw commented 3 months ago

The LLM database schema for storing responses needs to evolve to support some new features:

"Mock" responses. Sometimes it's useful to run a prompt that contains several previous messages which didn't actually come from a model - for example, prompting Claude 3 Haiku with a bunch of examples that are input as previous user messages. Differentiating these from actual text returned by a model is important to support LLM's goal of recording responses from models over time so you can analyze them later.
Another Claude feature: prompt caching. This is a larger feature to decide how to build, but since one of the uses of prompt caching is for longer conversations it's important to be able to remember at what point in the conversation the "cache_control": {"type": "ephemeral"} rule was injected.
Images! For prompts that use images one of the following three needs to be stored: the image URL (for images that were referenced by their URL) the image file path (for images that were a file on disk) or the raw bytes of the image itself (for cases where we want that stored in the database as opposed to living separately on disk)
Other multi-modal data: Gemini Pro accepts PDF and audio and video, fer example
That Claude trick where you can pre-fill the prefix for the response, passing <!doctype html><html> to ensure an HTML response for example. If that trick is used it should be recorded in the database as well.

simonw commented 3 months ago

Current schema: https://github.com/simonw/llm/blob/e867e13d1bb06d14fca5176e8bfe4dae2db000a3/docs/logging.md#L146-L168

simonw commented 3 months ago

Since some of these features are specific to Claude, it may make sense to have some kind of JSON column that specific LLM plugins can use to record information that only they care about - maybe using JSON keys with their name in e.g. "llm-claude-3:prefix": "<!doctype html><html>".

simonw commented 3 months ago

Does it even make sense for a conversation that was "real" (user submits a prompt, gets a model response, sends a reply, gets another model response) to be stored in the same schema as a conversation with simulated aspects (user sends an example-prompt/example-response/example-prompt/example-resoponse/real-prompt sequence)?

I think it does, especially since any follow-up prompts after that will still send the full sequence that's been stored just as it would if it wasn't full of mocked data.

Terminology here is hard. Are these "fake" responses? "mock" responses? Something else?

simonw commented 3 months ago

Worth considering tools here too, since those may require special thought about how to construct the schema.

simonw commented 3 months ago

I think "extras" might be a good way to model extra things that are specific to certain models. I could have a extras_json column for storing these. For places where a model needs to be passed extra custom arguments I could use -x/--extra key value in the llm prompt CLI options.

Not sure they should have the same name though, the extras_json column may store things that aren't related to -x/--extra options so perhaps they should be different.

plugin_json is another option here, for JSON that is specific to the LLM plugin used for the prompt.

simonw commented 3 months ago

Here's a schema consideration from tool use with Claude - a response might come back like this: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#example-api-response-with-a-tool-use-content-block

{
  "id": "msg_01Aq9w938a90dw8q",
  "model": "claude-3-5-sonnet-20240620",
  "stop_reason": "tool_use",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "<thinking>I need to use the get_weather, and the user wants SF, which is likely San Francisco, CA.</thinking>"
    },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": {"location": "San Francisco, CA", "unit": "celsius"}
    }
  ]
}

How should that be stored in the database? It's a single response from the assistant but it has two items in content - one of type text and one of type tool_use.

Also from that section of documentation:

Unlike APIs that separate tool use or use special roles like tool or function, Anthropic's API integrates tools directly into the user and assistant message structure.

Messages contain arrays of text, image, tool_use, and tool_result blocks. user messages include client-side content and tool_result, while assistant messages contain AI-generated content and tool_use.

simonw / llm

Design new LLM database schema #556