GUI for monitoring and view progress of task

nhha1602 commented 2 months ago

Self Checks

[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Do we support a GUI that we can see the progress of task processing or monitor of task processing on worker ? It is helpful for the task that take time to handle when we use API to add file for a Knowledge ...

2. Additional context or comments

No response

3. Can you help us with this feature?

[ ] I am interested in contributing to this feature.

dosubot[bot] commented 2 months ago

I found a similar unsolved discussion: UX Chat Page UI: Add visibility of response status for each chat at the chatpage level ^[1].

Regarding your question, the Dify platform does have built-in monitoring tools and interfaces for tracking task progress. It includes "observability features" and "LLMOps" which allow you to monitor and analyze application logs and performance over time. Additionally, Dify can be integrated with external Ops tools like LangSmith and Langfuse to provide comprehensive tracking and deep evaluation capabilities throughout the application lifecycle, from prototyping to production ^[2]^[3]^[4]^[5].

For task status or progress information, Dify provides an API endpoint to get the document embedding status (progress):

curl --location --request GET 'https://api.dify.ai/v1/datasets/{dataset_id}/documents/{batch}/indexing-status' \
--header 'Authorization: Bearer {api_key}'

Additionally, Dify offers a comprehensive set of dataset management APIs that allow you to create, list, and manage datasets and documents, as well as retrieve their statuses ^[6]^[7].

For long-running tasks, Dify handles and monitors them through real-time monitoring of performance data, ensuring stable operation of applications and providing complete log records ^[5].

There is also existing documentation and code related to task management and monitoring in the Dify codebase. The documentation mentions the Worker Asynchronous Queue Consumption Service and provides commands to start the Worker service for consuming asynchronous tasks from the queue ^[8]^[9]^[2]^[10].

Lastly, Dify recommends integrating with third-party LLMOps tools like LangSmith and Langfuse for monitoring and tracking task progress. These tools provide comprehensive tracking and deep evaluation capabilities for LLM applications ^[3]^[11].

To continue talking to Dosu, mention @dosu.

crazywoola commented 2 months ago

I don't think it's a feature request to Dify itself. Try this one. https://github.com/mher/flower

langgenius / dify