Add Interactive Action Buttons

[sergesha]

Proposal - #1764 Add Interactive Action Buttons

Description

Introduce the possibility to create and manage interactive buttons within the chat user interface. Such buttons would allow users to perform specific actions by clicking them. From an implementation perspective, such an action button would be linked to an arbitrary callback function in the backend. See more thoughts on implementation below.

Inspired by the Chainlit Actions concept.

Use case/motivation

• From a user perspective, this could make some types of interactions more intuitive and efficient:
  • Intuitive when users expect to be presented with a set of options and just having to click them, not having to type somehting out on the keyboard.
  • Efficient when performing repetitive tasks such as sending common starter messages.
• From a developer perspective, this can help to implement more predictable "conversational" flows by giving the user a predetermined set of options to choose from rather than relying on a free text response.

Thoughts on UI

Placement of buttons

There are at least two "levels" of actions:

• Message Level: Actions that relate to a particular message, either "any message" or the "last message only":
  • predetermined actions in response to an assistant message,
  • can be disabled (or removed) when clicked or outdated

    

• Conversation Level: Actions that relate to an entire conversation:
  • on top of chat input field,
  • constantly evolves, like a MacBook Pro touch bar,
  • can be used for suggested questions or predetermined actions in response to an assistant message or many other things

    

  • At the moment, it's still unclear how to handle this because the DIAL SDK and DIAL App do not manage the conversations directly; this logic resides within DIAL Core, right :question:

Our suggestion is to start with message-level actions since the implementation is more clear there. However, we may want to discuss conversation level actions as well so we can choose an initial implementation that is easier to extend later.

Sizing, text overflow

Buttons should expand in size to hold larger text, e.g.:

• Good: Ask "How can I do this thing?"
• Bad: Ask "How ca...

Action types (out of scope)

A future idea that could easily come on top of this and may inform current implementation: Different types of action buttons to indicate expected behavior, marked by dedicated icons and a meaningful tooltip.

• Message type
  • 💬 "Is DIAL fully open source?"
  • "Click to send the message displayed on the button."
• Link type:
  • 🔗 epam-rail.com
  • "Click to open https://epam-rail.com/news/2023-12-13/epam-announce"
• Form submission
  • 📝 Share your feedback
  • "Click to open the user satisfaction survey"
• etc.

The same look can be implemented without a dedicated feature for this, but standardizing the types would create a more unified user experience across various DIAL apps.

Thoughts on back-end

Chainlit uses WebSockets which facilitates easy back-and-forth sending of actions.
Since DIAL does not use WebSockets, we cannot take inspiration from Chainlit and need a different solution.

We have briefly explored various implementation possibilities and were able to make most of them work.
Before we invest more time, we would like to discuss these possibilities with the DIAL team to ensure alignment with the overall vision.

Sending "Actions" Definitions from DIAL App to DIAL Chat

Option 1: Using custom_content.state and Method set_state

• Define actions within the custom_content.state object and use the set_state method to store this definition.

   from typing import Dict, Optional, Any, List
   from aidial_sdk.pydantic_v1 import BaseModel, Field
   from aidial_sdk.chat_completion import Choice
  
   # This types (ArgumentSchema and Action) might be introduced in DIAL App only, no need to change DIAL SDK
   class ArgumentSchema(BaseModel):
       type: str = Field(..., description="The type of the argument, e.g., 'string', 'number'")
       description: Optional[str] = Field(None, description="A brief description of the argument")
       default: Optional[Any] = Field(None, description="The default value of the argument, if any")
       value: Optional[Any] = Field(None, description="The actual value of the argument, when set")
  
   class Action(BaseModel):
       name: str = Field(..., description="The name of the action button")
       actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
       args: Optional[List[ArgumentSchema]] = Field(None, description="List of arguments required to call the action")     
       options: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Additional parameters for the action")
  
   def add_actions(self, choice: Choice):
       choice.set_state({
           'actions': [
               {"name": "Action Button 1", 'actionId': "action-id-1", 'options': {"key1": "value1", "key2": "value2"}},
               {"name": "Action Button 2", 'actionId': "action-id-2", 'args': [
                   {"type": "string", "description": "The user input", "default": "default value"},
                   {"type": "number", "description": "The amount", "default": 10}
               ]}
           ]
       })

• Pros: We might consider available "actions" as a kind of state for a message. And it works.

• Cons: Using state instead of a dedicated place for actions makes them seem like an afterthought and not a properly integrated concept.

Option 2: Extending Choice (entity from aidial_sdk.chat_completion)

• Add a separate actions field to the Choice class to store action definitions directly.

   from typing import Dict, Optional, Any, List
   from aidial_sdk.pydantic_v1 import BaseModel, Field 
  
   class ArgumentSchema(BaseModel):
        type: str = Field(..., description="The type of the argument, e.g., 'string', 'number'")
        description: Optional[str] = Field(None, description="A brief description of the argument")
        default: Optional[Any] = Field(None, description="The default value of the argument, if any")
        value: Optional[Any] = Field(None, description="The actual value of the argument, when set")
  
   class Action(BaseModel):
        name: str = Field(..., description="The name of the action button")
        actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
        args: Optional[List[ArgumentSchema]] = Field(None, description="List of arguments required to call the action")     
        options: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Additional parameters for the action")
  
   class Choice(BaseModel):
        actions: Optional[List[Action]] = Field(None, description="List of action buttons")

• Pros: Provides a proper dedicated place for actions.
• Cons: Necessitates making changes to the DIAL SDK.

Calling "Actions" from DIAL Chat to DIAL App

Option 1: Using custom_content.state

• Send a ChatCompletionRequest containing, for example, the actionId to be handled by the DIAL App through the chat_completion method.
• Pros: Using the same "container" for communicating with backend both ways (receiving list of actions/sending action call)
• Cons:
  • May be considered a hack and could lead to confusion.
  • Also, new questions arise:
    • should we send this along with the 'user', 'system' or 'action' message (introducing a new role: "action")
    • should we leave the input empty or not?

Option 2: Using custom_fields

• Send a ChatCompletionRequest containing the actionId in custom_fields to be handled by the DIAL App through the chat_completion method.

   from typing import Any, Dict, List, Optional
   from aidial_sdk.pydantic_v1 import BaseModel, PositiveInt, Field
   from aidial_sdk.chat_completion.request import Addon, AzureChatCompletionRequest, ExtraForbidModel
  
   class Action(BaseModel):
      name: str = Field(..., description="The name of the action button")
      actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
  
   class ChatCompletionRequestCustomFields(ExtraForbidModel):
       configuration: Optional[Dict[str, Any]] = None
       actions: Optional[List[Action]] = Field(None, description="List of action buttons")
  
   class ChatCompletionRequest(AzureChatCompletionRequest):
       addons: Optional[List[Addon]] = None
       max_prompt_tokens: Optional[PositiveInt] = None
       custom_fields: Optional[ChatCompletionRequestCustomFields] = None

• Pros: Utilizes a field that, by design, appears to be intended for this purpose.
• Cons:
  • May be considered a hack and could lead to confusion.
  • Necessitates changing the DIAL SDK to extend ChatCompletionRequestCustomFields.

Option 3: Extending DIAL SDK

• Introduce ActionRequest:

   from aidial_sdk.pydantic_v1 import Field, StrictStr
   from aidial_sdk.deployment.from_request_mixin import FromRequestBasicMixin
  
   class ActionRequest(FromRequestBasicMixin):
      action_id: StrictStr = Field(None, alias="actionId")

• Add New Route in application.py: /openai/deployments/{deployment_id}/action.

• Extend ChatCompletion to Add action_response:

   from typing import Any
   from aidial_sdk.deployment.action import ActionRequest
  
   class ChatCompletion:
       # ...
  
       async def action_response(self, request: ActionRequest) -> Any:
           """Implement action response logic"""

• Pros: More solid and permanent.

• Cons: Necessitates making changes to the DIAL SDK.

Overall, it seems that for a proper implementation, changes are needed for the DIAL SDK and potentially DIAL CORE as well, as represented by the last option in each category.

Related issues

No response

Confidential information

• [x] I confirm that do not share any confidential information

[sergesha]

Here is a demo application, with actions implemented: https://github.com/WEAIDiscovery/dial-app-actions-demo