glacierck
commented
2 hours ago If it supports creating new queues and the worker processes indexing tasks evenly across multiple queues. That can also solve the problem of indexing tasks with different priorities in the enterprise system mentioned above.
It is possible to consider setting up multiple task queues and sending them all to one queue by default. By controlling the parameters, tasks can be pushed to other queues, which can effectively solve the blocking caused by a single queue and affect the priority of index tasks.
This is a load balancing solution for indexing tasks across multiple queues.
Self Checks
1. Is this request related to a challenge you're experiencing? Tell me about your story.
My application scenario requires frequent updates to individual documents, which requires timely index updates. But if the current task queue is very long (1000 new documents imported from a batch), it will block the current queue. At this point, the priority of re indexing a single document is significantly higher than that of batch indexing, and it needs to be queued to the front of the queue. I hope this queue jumping feature can be reflected in the API：
/datasets/{dataset_id}/document/create_by_file2. Additional context or comments
There are many scenarios that require queue priority indexing, such as prioritizing the processing of documents for users with high permissions and prioritizing the processing of documents with high importance levels, which proves that queue jumping is essential.
3. Can you help us with this feature?