Ability to "tag" invocations of the tool

[simonw]

I want to see logs just of prompts run by the tool I describe here: https://til.simonwillison.net/llms/claude-hacker-news-themes

It shells out to LLM. It would be neat if it could do this:

curl ... | llm -m claude 'Summarize the themes of the opinions expressed here, including quotes (with author attribution) where appropriate.' \
  --tag hn-summary

That --tag gets persisted in the database.

Then this could return logs with that tag:

llm logs --tag hn-summary

[simonw]

Do tags apply to individual prompt responses or to whole conversations?

I think they are recorded at the response level - but it would be useful to be able to request logs of conversations featuring a specific tag where that tag could be attached to any of the responses in that conversation.

Maybe this:

llm logs --ctag hn-summary

Or a better name.

[simonw]

Or maybe this:

llm logs \
  --tag hn-summary --conversations

Which means: find any responses matching this search criteria, but return the full conversation they are a part of.

This could work for other search operators as well. The logs tool would always default to operating on individual responses but could be expanded to return the full conversation.

[simonw]

Potential alternative to tags: Flickr style machine tag pairs, or even just arbitrary JSON metadata.

[simonw]

AWS tags are key value pairs: https://docs.aws.amazon.com/tag-editor/latest/userguide/tagging.html

llm "prompt" --tag tool hn-summary

Same syntax for searches as well.

I'm not sure about this, feels non-obvious. Useful data model though.

[simonw]

Or a weird middle ground:

llm "prompt" --tag tool:hn-summary

So tags are still single values, but there's a naming convention for doing key/value tags. I actually quite like this - it's self-documenting:

Could have a search operation for tags that match wildcards to help run searches.

Maybe disallow tags that end in * to reserve that character for searching.

Or have a rule that a tag cannot end in a colon with nothing following it, then tool: can be a valid search operation without having to teach people that * needs to be single quoted in their shell.