Introduce the possibility to create and manage interactive buttons within the chat user interface.
Such buttons would allow users to perform specific actions by clicking them.
From an implementation perspective, such an action button would be linked to an arbitrary callback function in the backend. See more thoughts on implementation below.
From a user perspective, this could make some types of interactions more intuitive and efficient:
Intuitive when users expect to be presented with a set of options and just having to click them, not having to type somehting out on the keyboard.
Efficient when performing repetitive tasks such as sending common starter messages.
From a developer perspective, this can help to implement more predictable "conversational" flows by giving the user a predetermined set of options to choose from rather than relying on a free text response.
Thoughts on UI
Placement of buttons
There are at least two "levels" of actions:
Message Level: Actions that relate to a particular message, either "any message" or the "last message only":
predetermined actions in response to an assistant message,
can be disabled (or removed) when clicked or outdated
Conversation Level: Actions that relate to an entire conversation:
on top of chat input field,
constantly evolves, like a MacBook Pro touch bar,
can be used for suggested questions or predetermined actions in response to an assistant message or many other things
At the moment, it's still unclear how to handle this because the DIAL SDK and DIAL App do not manage the conversations directly;
this logic resides within DIAL Core, right
:question:
Our suggestion is to start with message-level actions since the implementation is more clear there. However, we may want to discuss conversation level actions as well so we can choose an initial implementation that is easier to extend later.
Sizing, text overflow
Buttons should expand in size to hold larger text, e.g.:
Good: Ask "How can I do this thing?"
Bad: Ask "How ca...
Action types (out of scope)
A future idea that could easily come on top of this and may inform current implementation: Different types of action buttons to indicate expected behavior, marked by dedicated icons and a meaningful tooltip.
Message type
💬 "Is DIAL fully open source?"
"Click to send the message displayed on the button."
The same look can be implemented without a dedicated feature for this, but standardizing the types would create a more unified user experience across various DIAL apps.
Thoughts on back-end
Chainlit uses WebSockets which facilitates easy back-and-forth sending of actions.
Since DIAL does not use WebSockets, we cannot take inspiration from Chainlit and need a different solution.
We have briefly explored various implementation possibilities and were able to make most of them work.
Before we invest more time, we would like to discuss these possibilities with the DIAL team to ensure alignment with the overall vision.
Sending "Actions" Definitions from DIAL App to DIAL Chat
Option 1: Using custom_content.state and Method set_state
Define actions within the custom_content.state object and use the set_state method to store this definition.
from typing import Dict, Optional, Any, List
from aidial_sdk.pydantic_v1 import BaseModel, Field
from aidial_sdk.chat_completion import Choice
# This types (ArgumentSchema and Action) might be introduced in DIAL App only, no need to change DIAL SDK
class ArgumentSchema(BaseModel):
type: str = Field(..., description="The type of the argument, e.g., 'string', 'number'")
description: Optional[str] = Field(None, description="A brief description of the argument")
default: Optional[Any] = Field(None, description="The default value of the argument, if any")
value: Optional[Any] = Field(None, description="The actual value of the argument, when set")
class Action(BaseModel):
name: str = Field(..., description="The name of the action button")
actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
args: Optional[List[ArgumentSchema]] = Field(None, description="List of arguments required to call the action")
options: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Additional parameters for the action")
def add_actions(self, choice: Choice):
choice.set_state({
'actions': [
{"name": "Action Button 1", 'actionId': "action-id-1", 'options': {"key1": "value1", "key2": "value2"}},
{"name": "Action Button 2", 'actionId': "action-id-2", 'args': [
{"type": "string", "description": "The user input", "default": "default value"},
{"type": "number", "description": "The amount", "default": 10}
]}
]
})
Pros: We might consider available "actions" as a kind of state for a message. And it works.
Cons: Using state instead of a dedicated place for actions makes them seem like an afterthought and not a properly integrated concept.
Option 2: Extending Choice (entity from aidial_sdk.chat_completion)
Add a separate actions field to the Choice class to store action definitions directly.
from typing import Dict, Optional, Any, List
from aidial_sdk.pydantic_v1 import BaseModel, Field
class ArgumentSchema(BaseModel):
type: str = Field(..., description="The type of the argument, e.g., 'string', 'number'")
description: Optional[str] = Field(None, description="A brief description of the argument")
default: Optional[Any] = Field(None, description="The default value of the argument, if any")
value: Optional[Any] = Field(None, description="The actual value of the argument, when set")
class Action(BaseModel):
name: str = Field(..., description="The name of the action button")
actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
args: Optional[List[ArgumentSchema]] = Field(None, description="List of arguments required to call the action")
options: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Additional parameters for the action")
class Choice(BaseModel):
actions: Optional[List[Action]] = Field(None, description="List of action buttons")
Pros: Provides a proper dedicated place for actions.
Cons: Necessitates making changes to the DIAL SDK.
Calling "Actions" from DIAL Chat to DIAL App
Option 1: Using custom_content.state
Send a ChatCompletionRequest containing, for example, the actionId to be handled by the DIAL App through the chat_completion method.
Pros: Using the same "container" for communicating with backend both ways (receiving list of actions/sending action call)
Cons:
May be considered a hack and could lead to confusion.
Also, new questions arise:
should we send this along with the 'user', 'system' or 'action' message (introducing a new role: "action")
should we leave the input empty or not?
Option 2: Using custom_fields
Send a ChatCompletionRequest containing the actionId in custom_fields to be handled by the DIAL App through the chat_completion method.
from typing import Any, Dict, List, Optional
from aidial_sdk.pydantic_v1 import BaseModel, PositiveInt, Field
from aidial_sdk.chat_completion.request import Addon, AzureChatCompletionRequest, ExtraForbidModel
class Action(BaseModel):
name: str = Field(..., description="The name of the action button")
actionId: str = Field(..., alias="actionId", description="The unique identifier for the action")
class ChatCompletionRequestCustomFields(ExtraForbidModel):
configuration: Optional[Dict[str, Any]] = None
actions: Optional[List[Action]] = Field(None, description="List of action buttons")
class ChatCompletionRequest(AzureChatCompletionRequest):
addons: Optional[List[Addon]] = None
max_prompt_tokens: Optional[PositiveInt] = None
custom_fields: Optional[ChatCompletionRequestCustomFields] = None
Pros: Utilizes a field that, by design, appears to be intended for this purpose.
Cons:
May be considered a hack and could lead to confusion.
Necessitates changing the DIAL SDK to extend ChatCompletionRequestCustomFields.
Option 3: Extending DIAL SDK
Introduce ActionRequest:
from aidial_sdk.pydantic_v1 import Field, StrictStr
from aidial_sdk.deployment.from_request_mixin import FromRequestBasicMixin
class ActionRequest(FromRequestBasicMixin):
action_id: StrictStr = Field(None, alias="actionId")
Add New Route in application.py:
/openai/deployments/{deployment_id}/action.
Extend ChatCompletion to Add action_response:
from typing import Any
from aidial_sdk.deployment.action import ActionRequest
class ChatCompletion:
# ...
async def action_response(self, request: ActionRequest) -> Any:
"""Implement action response logic"""
Pros: More solid and permanent.
Cons: Necessitates making changes to the DIAL SDK.
Overall, it seems that for a proper implementation, changes are needed for the DIAL SDK and potentially DIAL CORE as well, as represented by the last option in each category.
Related issues
No response
Confidential information
[x] I confirm that do not share any confidential information
Proposal - #1764 Add Interactive Action Buttons
Description
Introduce the possibility to create and manage interactive buttons within the chat user interface. Such buttons would allow users to perform specific actions by clicking them. From an implementation perspective, such an action button would be linked to an arbitrary callback function in the backend. See more thoughts on implementation below.
Inspired by the Chainlit Actions concept.
Use case/motivation
Thoughts on UI
Placement of buttons
There are at least two "levels" of actions:
Our suggestion is to start with message-level actions since the implementation is more clear there. However, we may want to discuss conversation level actions as well so we can choose an initial implementation that is easier to extend later.
Sizing, text overflow
Buttons should expand in size to hold larger text, e.g.:
Ask "How can I do this thing?"
Ask "How ca...
Action types (out of scope)
A future idea that could easily come on top of this and may inform current implementation: Different types of action buttons to indicate expected behavior, marked by dedicated icons and a meaningful tooltip.
💬 "Is DIAL fully open source?"
🔗 epam-rail.com
📝 Share your feedback
The same look can be implemented without a dedicated feature for this, but standardizing the types would create a more unified user experience across various DIAL apps.
Thoughts on back-end
Chainlit uses WebSockets which facilitates easy back-and-forth sending of actions.
Since DIAL does not use WebSockets, we cannot take inspiration from Chainlit and need a different solution.
We have briefly explored various implementation possibilities and were able to make most of them work.
Before we invest more time, we would like to discuss these possibilities with the DIAL team to ensure alignment with the overall vision.
Sending "Actions" Definitions from DIAL App to DIAL Chat
Option 1: Using
custom_content.state
and Methodset_state
Define
actions
within thecustom_content.state
object and use theset_state
method to store this definition.Pros: We might consider available "actions" as a kind of state for a message. And it works.
Cons: Using state instead of a dedicated place for actions makes them seem like an afterthought and not a properly integrated concept.
Option 2: Extending
Choice
(entity from aidial_sdk.chat_completion)Add a separate
actions
field to theChoice
class to store action definitions directly.Calling "Actions" from DIAL Chat to DIAL App
Option 1: Using
custom_content.state
ChatCompletionRequest
containing, for example, theactionId
to be handled by the DIAL App through thechat_completion
method.Option 2: Using
custom_fields
Send a
ChatCompletionRequest
containing theactionId
incustom_fields
to be handled by the DIAL App through thechat_completion
method.ChatCompletionRequestCustomFields
.Option 3: Extending DIAL SDK
Introduce
ActionRequest
:Add New Route in
application.py
:/openai/deployments/{deployment_id}/action
.Extend
ChatCompletion
to Addaction_response
:Pros: More solid and permanent.
Cons: Necessitates making changes to the DIAL SDK.
Overall, it seems that for a proper implementation, changes are needed for the DIAL SDK and potentially DIAL CORE as well, as represented by the last option in each category.
Related issues
No response
Confidential information