ahuang11
commented
1 month ago I looked up all llm.invoke and I think I want the prompts to be structured like:
prompts/
- assistant:
- check_validity.jinja2
- handle_exception.jinja2
- source_agent.jinja2
- chat_agent.jinja2
- chat_details_agent.jinja2
- table_list_agent.jinja2
- sql_agent.jinja2
- hvplot_agent.jinja2
- analysis_agent.jinja2
- vega_lite_agent.jinja2
- source_agent:
- n/a
- chat_agent:
- require_data.jinja2
- system.jinja2
- chat_details_agent:
- choose_topic.jinja2
- system.jinja2
- table_list_agent:
- n/a
- sql_agent:
- require_join.jinja2
- select_tables.jinja2
- system.jinja2
- hvplot_agent:
- fill_plot.jinja2
- system.jinja2
- analysis_agent:
- system.jinja2
- vega_lite_agent:
- system.jinja2I think each agent should be refactored with params like:
class ChatAgent(Agent):
purpose = param.String(
default=(
"Chats and provides info about high level data related topics, "
"e.g. what datasets are available, the columns of the data or "
"statistics about the data, and continuing the conversation. "
"Is capable of providing suggestions to get started or comment on interesting tidbits. "
"If data is available, it can also talk about the data itself."
),
doc="The directive that the Assistant will use to determine whether this agent should be selected."
)
system_prompt = param.String(
default=(
"Act as a helpful assistant for high-level data exploration, focusing on available datasets and, "
"only if data is available, explaining the purpose of each column. Offer suggestions for getting "
"started if needed, remaining factual and avoiding speculation."
)
)
prompts = param.Dict(default={"require_data": "Assess if the user's prompt requires loading data."})
response_model = param.ClassSelector(class_=BaseModel, is_instance=False)
requires = param.List(default=["current_source"], readonly=True)It can also be paths:
class ChatAgent(Agent):
purpose = param.String(
default=str(THIS_DIR / "prompts" / "assistant" / "chat_agent.jinja2")
doc="The directive that the Assistant will use to determine whether this agent should be selected."
)
system_prompt = param.String(
default=str(THIS_DIR / "prompts" / "chat_agent" / "system.jinja2")
)
prompts = param.Dict(default={"require_data": "str(THIS_DIR / "prompts" / "chat_agent" / "require_data.jinja2")
"})
response_model = param.ClassSelector(class_=BaseModel, is_instance=False)
requires = param.List(default=["current_source"], readonly=True)So that users can easily modify the prompts by passing a string:
ChatAgent(purpose="Choose this for chatting.", system_prompt="Talk about the data like a pirate")Or path:
ChatAgent(purpose="path/to/assistant/chat_agent.jinja2", system_prompt="path/to/chat_agent/system.jinja2")It would also be nice to have a Prompt parameterized config class that contains all the built-in assistant/agents prompts, and users can use it to explore built-in prompts and update it.
class PromptsConfig(param.Parameterized):
def __init__(self, **params):
super().__init__(**params)
"""Recursively reads prompts directory and outputs a dict."""
self._prompts = ...
def dict(self) -> dict:
return self._prompts
def update(self, key: str, value: str) -> dict:
"""Updates a specific key, delimited by `.`, e.g. assistant.chat_agent or chat_agent.system"""
self._prompts[...] = value
Also, I'm starting to wonder whether each agent should only have, at most, a single invoke, i.e. split out SQLAgent into JoinAgent and QueryAgent, so that there's no need for the prompts param.
philippjfr
At the current moment, there are multiple entry points for prompting the LLM
BaseModelsBaseModelsI probably am missing one, but I propose that for maximum customizability, we should migrate all of these to jinja2 templates, and dynamically read the prompts from the jinja2 templates and create models dynamically too.
That way, users can update the prompts to their liking by either passing in
system_prompt_pathswith the corresponding names or updating aSYSTEM_PROMPTS[key]dict.We should also make it easy for users to discover these prompts.