Task Queue

[movchan74]

Summary: This update introduces a task queue system to the project, enabling deferred execution of endpoints. This feature allows users to submit multiple tasks for background processing, which is particularly beneficial for long-running operations like indexing.

Key Changes:

1. Task Entity in SQL Integration: Added a new task entity to the SQL database for managing queued tasks.
2. Task Repository: Implemented a repository for handling task-related database operations.
3. Defer Option for Endpoints: Introduced an optional defer parameter for endpoints, defaulting to false, to allow tasks to be processed later.
4. Request Worker Update: Updated the request worker with a call_endpoint method, enabling execution of endpoints through Ray remote calls instead of HTTP.
5. Task Queue Deployment: Introduced a deployment that implements a task queue system that fetches tasks from the SQL table and uses the request worker to execute them.
6. Task Queue Settings: Added configuration settings for managing the task queue.
7. Integration with AanaSDK: Integrated the task queue into the AanaSDK class for automatic deployment.
8. Task Queue Tests: Implemented tests to ensure the functionality and reliability of the task queue system.

[movchan74]

Task Processing Time Measurement

Dummy App (Returns Lowercase Text)

To measure the overhead of processing a task from a queue, a dummy application was used. This application simply returns lowercase text. Two request handlers were consistently used for these measurements.

Setup	Workers	Time per Query (ms)
Without Queue	N/A	12
SQLite Queue	2	33
SQLite Queue	4	31
PostgreSQL Queue	4	21

Observation: The overhead introduced by using a task queue is noticeable. However, increasing the number of workers beyond 4 does not significantly reduce the processing time.

Heavier Task (100 ms Sleep Added to Simulate Real Workload)

To simulate a more realistic workload, a task was modified to include a 100 ms sleep. This adjustment was made to better understand the queue's impact under a more demanding scenario.

Setup	Workers	Time per Query (ms)
Without Queue	N/A	101.81
SQLite Queue	4	103.57
PostgreSQL Queue	4	102.12

Observation: When simulating a real workload with a 100 ms sleep, the difference between using a queue and processing tasks without a queue becomes minimal. The overhead introduced by the queue is less significant compared to the total processing time of the task.

Conclusion: While queues introduce some overhead in low-latency scenarios, their impact is substantially reduced under heavier workloads.