Open wch opened 2 weeks ago
Thanks. Out of curiosity, did you have to implement a new strategy, or is #1755 enough to accomplish what you need?
{ "role": "user", "content": "What color is the sky?" }
The input needs this format:
{
"role": "user",
"content": [
{
"type": "text",
"text": "What color is the sky?",
"cache_control": {"type": "ephemeral"}
}
]
}
So for conversations, after calling `chat.messages(format="anthropic")), the result needs to be transformed to the new format before sending it to the API endpoint. And the system prompt needs to be transformed in the same way.
I wrote a function to do the transformation here, but note that it won't cover all cases -- for example, if you use non-text inputs, it will throw an error. It is sufficient for the my use case though:
So we should think about how to make prompt caching easier to use.
Good to know. As a heads up, we're soon going to promote chatlas as the way to manage the conversation state (and generate responses) instead of chat.message(format="...")
. That said, we don't currently have a good way to take advantage of Anthropic's approach to prompt caching, so probably worth sticking with chat.message(format="anthropic")
in shiny assistant, at least for now.
Previously, the error message said this:
But the function
shiny.ui._chat_types.registry.register()
doesn't exist. I changed it to what I think is the correct function,shiny.ui._chat_normalize.message_normalizer_registry.register()
, but please confirm that it's right.