posit-dev / py-shiny

Shiny for Python
https://shiny.posit.co/py/
MIT License
1.32k stars 81 forks source link

Change chat custom normalizer error message to point to correct function #1754

Open wch opened 2 weeks ago

wch commented 2 weeks ago

Previously, the error message said this:

Consider registering a custom normalizer via shiny.ui._chat_types.registry.register()

But the function shiny.ui._chat_types.registry.register() doesn't exist. I changed it to what I think is the correct function, shiny.ui._chat_normalize.message_normalizer_registry.register(), but please confirm that it's right.

cpsievert commented 2 weeks ago

Thanks. Out of curiosity, did you have to implement a new strategy, or is #1755 enough to accomplish what you need?

wch commented 2 weeks ago

1755 was enough for my purposes. However, I should mention that the input message format when using prompt caching is different from the usual format. Instead of:

{ "role": "user", "content": "What color is the sky?" }

The input needs this format:

{
  "role": "user",
  "content": [
    {
      "type": "text",
      "text": "What color is the sky?",
      "cache_control": {"type": "ephemeral"}
    }
  ]
}

So for conversations, after calling `chat.messages(format="anthropic")), the result needs to be transformed to the new format before sending it to the API endpoint. And the system prompt needs to be transformed in the same way.

I wrote a function to do the transformation here, but note that it won't cover all cases -- for example, if you use non-text inputs, it will throw an error. It is sufficient for the my use case though:

https://github.com/posit-dev/shiny-assistant/blob/f383675d60c1e709d3c1bdd66962d7bc2d51f4e0/app.py#L552-L597

So we should think about how to make prompt caching easier to use.

cpsievert commented 2 weeks ago

Good to know. As a heads up, we're soon going to promote chatlas as the way to manage the conversation state (and generate responses) instead of chat.message(format="..."). That said, we don't currently have a good way to take advantage of Anthropic's approach to prompt caching, so probably worth sticking with chat.message(format="anthropic") in shiny assistant, at least for now.