Easily link `chat` object with a shinychat UI instance

[gadenbuie]

I'm not sure if this is primarily an elmer request or a shinychat request, but it'd be very nice if shinychat could be more tightly integrated with elmer.

In script or interactive environments, the elmer's chat$chat() interface is very friendly:

library(elmer)

chat <- elmer::chat_openai(model = "gpt-4o")

chat$chat(
  "What photographic choices were made here, and why do you think the photographer chose them?",
  content_image_file("photo.jpg")
)

chat$chat("Come up with an artsy, pretentious, minimalistic, abstract title for this photo.")

But when combined with Shiny there's at least the minimal boilerplate of hooking up an observer for input${id}_user_input, and the boilerplate picks up when you start doing some server-side interaction with the chat messages. Here's an example app where the above turns are handled via action buttons, but the user can still talk with the model

library(elmer)
library(shiny)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat"),
  actionButton("ask_photo", "Ask question with image"),
  actionButton("ask_follow_up", "Ask follow up question")
)

server <- function(input, output, session) {
  chat <- chat_openai(
    model = "gpt-4o",
  )

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })

  observeEvent(input$ask_photo, {
    turn <- Turn(
      "user",
      list(
        ContentText("What photographic choices were made here, and why do you think the photographer chose them?"),
        content_image_file("photo.jpg")
      )
    )

    chat_append_message(
      "chat",
      list(
        role = "user",
        content = turn@text,
      )
    )

    chat_append(
      "chat",
      chat$stream_async(!!!turn@contents)
    )
  })

  observeEvent(input$ask_follow_up, {
    q <- "Come up with an artsy, pretentious, minimalistic, abstract title for this photo."

    chat_append_message("chat", list(role = "user", content = q))
    chat_append("chat", chat$stream_async(q))
  })
}

shinyApp(ui, server)

I'm envisioning something like the following:

library(elmer)
library(shiny)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat"),
  actionButton("ask_photo", "Ask question with image"),
  actionButton("ask_follow_up", "Ask follow up question")
)

server <- function(input, output, session) {
  chat <- chat_openai(
    model = "gpt-4o",
  )

  # Connect elmer `chat` object with shinychat's "chat" UI
  shinychat::register_elmer_chat("chat", chat, append_method = "stream_async")

  # User input is handled automatically

  observeEvent(input$ask_photo, {
    # Constructed calls to `chat$chat()` are streamed to the UI automatically
    chat$chat(
      ContentText("What photographic choices were made here, and why do you think the photographer chose them?"),
      content_image_file("photo.jpg")
    )
  })

  observeEvent(input$ask_follow_up, {
    chat$chat(
      "Come up with an artsy, pretentious, minimalistic, abstract title for this photo."
    )
  })
}

shinyApp(ui, server)

In this scenario, some of the code would come from shinychat (setting up the observer) but there would also be changes in the Chat class to call shinychat's chat_append() or chat_append_messages(). That could be done through registering callbacks so the implementation could still live in shinychat and so other packages could extend the behavior.

That said, I could also see another scenario where the abstractions live mostly in shinychat and register_elmer_chat() associates the chat object with a particular chat UI ID and then new functions in shinychat append new turns to both chat and the UI.

[gadenbuie]

Just a follow up comment that I think most users would be well-served by a way to jointly update the UI and the actual chat, there are also scenarios where users might want to

1. Maintain separate turn history in the UI and the elmer chat instance.
   • Solution: Don't call the registration function to link the UI and chat instance.
2. Process the user input before sending it to the model.
   • Solution: Include an option to disable the automatic user input handling when registering.

[gadenbuie]

Another note: the more I think about this, the more I want an R6 interface for shinychat that takes an elmer chat object on init and has its own methods to do all of these things.

[hadley]

My feeling is that this should belong in shinychat, but I could be persuaded otherwise.

[gadenbuie]

I agree, @jcheng5 do you have the ability to move this issue to shinychat?

[hadley]

You can't move issues across orgs, unfortunately.

[gadenbuie]

Moved to https://github.com/jcheng5/shinychat/issues/14