Closed pranav-kural closed 1 month ago
On testing, observed that when defining an open-ended chat endpoint, query never gets sent to the LLM, due to an issue in the prompt. The prompt template being used didn't have the {{query}}
construct.
Also noticed warnings regarding prompt with certain name (e.g., openEndedSystemPrompt
) being over-written.
Fix for above issue added pre-release: #59
Checklist
Description
Add support for Multimodal I/O: Support multimodal input and output.
Impact (Why is this feature important?)
Will allow users to add media (images, videos, etc.) in input and get output that contains media and not just text responses.
Select Components this Feature will Impact
Select component(s) this feature will impact
Proposal (Optional)
Will require several changes to multiple components.
ChatAgent
class: should have a method that can handle multimodal input and generate multimodal output.defineChatEndpoint
to use the new method created inChatAgent
class to support multimodality.Could roll out the support for multimodal input-output for only chat endpoints not using the chat history and RAG, or just specify in the documentation that non-text information will not work with chat history and RAG.
Alternatives (Optional)
Can still use chat endpoints to generate multi-media content. Output will likely contain a URL to the generated content.
Can not provide multi-media content to models that support multi-modal input right now.
Resources (Optional)
Include any resources, references, or links that might be helpful in understanding or implementing this feature.