hjlarry commented 13 hours ago

Summary

[!Tip] Close issue syntax: Fixes #<issue number> or Resolves #<issue number>, see documentation for more details.

Currently, lots of LLM( gemini, sonnet ...) can directly process document, and make user's chat context based on these documents. This PR aimed to support this feature in a dify agent app. For the chatflow app, maybe this PR can resolve.

ChangeList

Backend

add DocumentPromptMessageContent and more model features, copied from https://github.com/langgenius/dify/pull/10679
support gemini series models receive document
add video feature to support video models, document feature to support document models

Frontend

add a switch button to control whether allow upload document
for not support video feature's LLM, open vision config will not allow upload video
chat page whether display file upload button depends on file.allowed_file_types has any value

remaining issues

the vision settings is strange, Resolution only affect Image type files, Upload Method and Upload Limit affect all type files.

Screenshots

Checklist

[!IMPORTANT]
Please review the checklist below before submitting your pull request.

[ ] This change requires a documentation update, included: Dify Document
[x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
[x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
[x] I've updated the documentation accordingly.
[x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

laipz8200 commented 9 hours ago

Thank you for this awesome contribution! Some changes in #10679 are still in processing, I'll review this PR after #10679 is merged.

laipz8200 commented 9 hours ago

the vision settings is strange, Resolution only affect Image type files, Upload Method and Upload Limit affect all type files.

I think this resolution config should be removed in the future.

laipz8200 commented 7 hours ago

Hi @hjlarry! #10679 is merged, could you please sync the code with the main branch?

hjlarry commented 7 hours ago

Hi @hjlarry! #10679 is merged, could you please sync the code with the main branch?

Done :)

laipz8200 commented 6 hours ago

https://github.com/user-attachments/assets/e4c34422-e458-4ddb-9e38-92cd72e1f350

hjlarry commented 6 hours ago

Screen.Recording.2024-11-22.at.5.56.05.PM.mov

seems the icon has been overwrite by the merge action, please try again

langgenius / dify

feat: support LLM process document file #10966