simonw / llm

Access large language models from the command-line
https://llm.datasette.io
Apache License 2.0
4.56k stars 252 forks source link

Design new LLM database schema #556

Open simonw opened 2 months ago

simonw commented 2 months ago

The LLM database schema for storing responses needs to evolve to support some new features:

simonw commented 2 months ago

Current schema: https://github.com/simonw/llm/blob/e867e13d1bb06d14fca5176e8bfe4dae2db000a3/docs/logging.md#L146-L168

simonw commented 2 months ago

Since some of these features are specific to Claude, it may make sense to have some kind of JSON column that specific LLM plugins can use to record information that only they care about - maybe using JSON keys with their name in e.g. "llm-claude-3:prefix": "<!doctype html><html>".

simonw commented 2 months ago

Does it even make sense for a conversation that was "real" (user submits a prompt, gets a model response, sends a reply, gets another model response) to be stored in the same schema as a conversation with simulated aspects (user sends an example-prompt/example-response/example-prompt/example-resoponse/real-prompt sequence)?

I think it does, especially since any follow-up prompts after that will still send the full sequence that's been stored just as it would if it wasn't full of mocked data.

Terminology here is hard. Are these "fake" responses? "mock" responses? Something else?

simonw commented 2 months ago

Worth considering tools here too, since those may require special thought about how to construct the schema.

simonw commented 2 months ago

I think "extras" might be a good way to model extra things that are specific to certain models. I could have a extras_json column for storing these. For places where a model needs to be passed extra custom arguments I could use -x/--extra key value in the llm prompt CLI options.

Not sure they should have the same name though, the extras_json column may store things that aren't related to -x/--extra options so perhaps they should be different.

plugin_json is another option here, for JSON that is specific to the LLM plugin used for the prompt.

simonw commented 2 months ago

Here's a schema consideration from tool use with Claude - a response might come back like this: https://docs.anthropic.com/en/docs/build-with-claude/tool-use#example-api-response-with-a-tool-use-content-block

{
  "id": "msg_01Aq9w938a90dw8q",
  "model": "claude-3-5-sonnet-20240620",
  "stop_reason": "tool_use",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "<thinking>I need to use the get_weather, and the user wants SF, which is likely San Francisco, CA.</thinking>"
    },
    {
      "type": "tool_use",
      "id": "toolu_01A09q90qw90lq917835lq9",
      "name": "get_weather",
      "input": {"location": "San Francisco, CA", "unit": "celsius"}
    }
  ]
}

How should that be stored in the database? It's a single response from the assistant but it has two items in content - one of type text and one of type tool_use.

Also from that section of documentation:

Unlike APIs that separate tool use or use special roles like tool or function, Anthropic's API integrates tools directly into the user and assistant message structure.

Messages contain arrays of text, image, tool_use, and tool_result blocks. user messages include client-side content and tool_result, while assistant messages contain AI-generated content and tool_use.