[ ] Document (or make easier) how to have Python code log to SQLite · Issue #228 · simonw/llm

Document (or make easier) how to have Python code log to SQLite

Snippet

Skip to content
Navigation Menu
simonw
/
llm

Type / to search
Code
Issues
160
Pull requests
42
Discussions
Actions
Projects
Security
Insights
Document (or make easier) how to have Python code log to SQLite #228
Open
simonw opened this issue  · 3 comments
Comments

Owner
simonw commented 
Question came up here: https://discord.com/channels/823971286308356157/1128504153841336370/1148406576739254363

Currently there isn't a documented way of having prompts run through the Python code get logged to SQLite.

The undocumented way is to do this:

db = sqlite_utils.Database(\"logs.db\")
response = model.prompt(\"prompt goes here\")
response.log_to_db(db)
But that's a) not documented and b) not much fun to have to remember to do every time.
  simonw added documentation  enhancement  python-api  labels 

Owner
Author
simonw commented 
I noticed this when I added the feature to https://github.com/simonw/llm-cluster that uses an LLM to generate a summary of each cluster - I would have liked to have captured my experiments with that tool, since it took some iterating on the prompt to get to something that worked OK.

Contributor
ealvar3z commented  • 
Hi @simonw,

I've been perusing around the issues to give back to the project and I came across this issue,. After reading your blog post regarding capturing experiments with the llm-cluster tool, which led me to the below possible solution.

This plugin ought to automatically log every prompt and its corresponding response to an SQLite database. Here's a 90% solution (I haven't written any tests yet). I simply followed the idioms in your llm-cluster plugin:

import sqlite_utils
from datetime import datetime

@llm.hookimpl
def register_commands(cli):
    @cli.command()
    @click.option(
        \"--log_db\",
        type=click.Path(
            file_okay=True, allow_dash=False, dir_okay=False, writable=True
        ),
        envvar=\"LLM_PROMPT_LOG_DB\",
        help='SQLite database file for logging')
    def log_prompt(log_db):
        db = sqlite_utils.Database(log_db or 'logs.db')
        db['logs'].create({'timestamp': str, 'prompt': str, 'response': str}, pk='timestamp', if_not_exists=True)

        def log_to_db(prompt, response):
            timestamp = datetime.now().isoformat()
            db['logs'].insert({'timestamp': timestamp, 'prompt': prompt, 'response': response})

        original_prompt = llm.get_model().prompt

        def new_prompt(user_prompt):
            response = original_prompt(user_prompt)
            log_to_db(user_prompt, response)
            return response

        llm.get_model().prompt = new_prompt
It simply adds a new CLI option --log_db to specify the SQLite database file for logging, overrides the prompt method of the language model to include the logging functionality, and ensures that every interaction with the model gets logged.

Best regards, and thank you for the llm project.

maxwelljoslyn commented  • 
I was looking for some way to log my conversation results from llm's Python API to a database -- the exact same way that the CLI version does, same schema and FTS, etc. I'll try the shortcut you've written in the first comment on this topic, and maybe I can help bring that into the Python API proper.

@ealvar3z's code looks like a good starting point as far as overriding the prompt method, but that's written as a plugin for the CLI, and not as an addition to the Python functionality of the library, which I think is warranted here. It could be a keyword option on model.prompt and model.conversation which causes requests/responses to get logged before being returned.

(What brought me here: yesterday I started working on a program that's a training ground for role-playing game Game Masters, which uses the LLMs as individual pseudo-players in an RPG group. All my LLM interactions are through your library as a Python package. So far, it feels Pythonic, comprehensive, and easy to use. Thank you for making it.)

Full Text

The issue "Document (or make easier) how to have Python code log to SQLite" was opened by simonw in the simonw/llm repository.

The issue description states that there is currently no documented way of having prompts run through the Python code get logged to SQLite. The undocumented way is to do the following:

db = sqlite_utils.Database("logs.db")
response = model.prompt("prompt goes here")
response.log_to_db(db)

However, this is not documented and it's "not much fun to have to remember to do every time".

The author, simonw, noticed this issue when adding a feature to the llm-cluster tool, where they would have liked to capture their experiments with the tool, as it took some iterating on the prompt to get to something that worked.

In the comments, ealvar3z provides a possible 90% solution in the form of a plugin. This plugin automatically logs every prompt and its corresponding response to an SQLite database. The plugin adds a new CLI option --log_db to specify the SQLite database file for logging, overrides the prompt method of the language model to include the logging functionality, and ensures that every interaction with the model gets logged.

maxwelljoslyn then comments that they were looking for a way to log their conversation results from the llm's Python API to a database, in the same way the CLI version does. They mention that ealvar3z's code looks like a good starting point, but that it's written as a plugin for the CLI, and not as an addition to the Python functionality of the library, which they think is warranted here. They suggest that it could be a keyword option on model.prompt and model.conversation which causes requests/responses to get logged before being returned.

irthomasthomas / undecidability