nasa-petal / bidara-deep-chat

BIDARA is a GPT-4o chatbot that was instructed to help scientists and engineers understand, learn from, and emulate the strategies used by living things to create sustainable designs and technologies using the Biomimicry Institute's step-by-step design process.
https://bit.ly/bidara-ai
Other
20 stars 5 forks source link

Support code_interpreter and vision for file uploads #179

Open bruffridge opened 1 month ago

bruffridge commented 1 month ago

Deep-chat 2.0 added support for v2 Assistants API. It also changed how uploaded files are handled. Now there is a files_tool_type parameter that determines which tool operates on file uploads. So instead of letting OpenAI decide all uploaded files are handled by the tool set by the parameter. This means ALL uploaded files can only be handled by one tool. We want it to be based on the file and what the user asks. For example, uploaded images should be passed to the gpt-4o for vision. Other files such as excel and csv the user might want visualized via code_interpreter. PDFs may want summarized via file_search.

https://github.com/OvidijusParsiunas/deep-chat/issues/190#issuecomment-2138206061