Claude

[josephjclark]

naive integration of Claude into the job_chat service

This should work seamlessly with Lightning, although the Lightning env will need updating to use an Anthropic API key (we may want to rename the var in the code base too)

At the moment the token usage is going to be pretty high - we're just throwing everything into the prompt and crossing our fingers.

There are other things I want to look into here but I may spin them out into other issues:

• [x] Put more stuff in the system prompt (are there benefits apart from caching?)
• [x] Use prompt caching for the system prompt (this is a whole thing) https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
  • adaptor APIs and the job writing guide would be good prompt candidates
• [x] Training the style of Claude's response. Verbose, terse, code only?
• [x] Careful use of examples
• [ ] Optimise long prompts with tags (maybe with rag?) https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/long-context-tips

Some other future work to consider:

• Use RAG to produce smaller inputs from the adaptor API. Re-using the search branch could give us good results here, resulting in smaller inputs and lower cost
• Use tools (functions which can call out to external services) https://docs.anthropic.com/en/docs/build-with-claude/tool-use. This basically means claude's response will be "hey call this tool for me", and you can send the result back to claude, which will then generate a final response. You could use it to call for a live weather forecast and then generate a prompt off the back of it. How could we use this?

[josephjclark]

This is disappointing:

Claude should refuse to answer a question like this, which is unrelated to openfn.

I'm also getting quickly annoyed by this obsequious customer service attitude

EDIT: I've re-worked the prompt to lean more heavily on the system prompt, and it seems to work better

[josephjclark]

I've plugged prompt caching in. I dunno.

• It costs +25% to write a prompt to the cache
• It only costs 10% to retrieve that prompt back to the cache
• The cache last 5 minutes (but is refreshed on read).

I am caching the job writing guide and adaptor doc stuff separately. I don't really know if that's the right thing to do.

I presume the cache is global, shared by all users (in the world). Because I guess all its doing is caching the embeddings from a raw string.

I have used a dollar of credit today just in testing, and I don't even think I've been testing very heavily. Yep, 40 calls. That's not a lot of traffic - but we send a LOT of input tokens because we include the adaptor API.

I think caching the job writing guide is a good idea. I think RAG for adaptor APIs is likely to be better because we can provide a significantly smaller, more relevant input.