search

OCA / queue

Asynchronous Job Queue
GNU Affero General Public License v3.0
179 stars 456 forks source link

[17.0][FIX] Refactor stop signal handling in QueueJobRunner #647

Open Maralai opened 5 months ago

Maralai commented 5 months ago

This commit addresses the issue: https://github.com/OCA/queue/issues/564.

Refactor QueueJobRunner to Enhance Performance, Signal Handling, and Stability

This pull request brings substantial improvements and new functionalities to the QueueJobRunner, which is focused on optimizing performance, signal handling, and stability. The significant changes include the transition to asynchronous HTTP requests, dynamic detection of new databases, and enhancements for robust signal handling.

Key Changes:

  1. Async HTTP Requests:

    • Transitioned from threading-based requests to asyncio and aiohttp for handling HTTP job runs asynchronously. This change enhances concurrency and network-bound operation efficiency.

  2. Socket Pair for Signal Handling:

    • Implemented socket pairs to effectively manage stop signals, ensuring compatibility across UNIX and Windows environments.

  3. Selectors for Notification Handling:

    • Adopted selectors.DefaultSelector() for handling database and stop signal notifications, improving efficiency and scalability of notification handling.

  4. Auto-Detection of New Databases:

    • Added functionality for periodically checking and initializing newly created databases with the queue_job module installed, allowing dynamic adaptation without server restart.

  5. Configurable Check Interval:

    • Introduced a parameter for setting the check interval for new databases, enabling a balance between responsiveness and system overhead.

  6. Database Keep-Alive Mechanism:

    • Added periodic SELECT 1 queries to keep database connections alive and prevent intermediaries from closing idle connections.

Please review the changes and provide feedback or approval for merging this enhanced version of the QueueJobRunner. These improvements address the aforementioned issue and the todo's included within the original file, increasing robustness, scalability, and overall performance. I have included test cases and attempted to comply with pre-commit tests, with the aim of achieving a green checkmark.

Thank you for your consideration.

OCA-git-bot commented 5 months ago

Hi @guewen, some modules you are maintaining are being modified, check this out!

github-actions[bot] commented 1 week ago

There hasn't been any activity on this pull request in the past 4 months, so it has been marked as stale and it will be closed automatically if no further activity occurs in the next 30 days. If you want this PR to never become stale, please ask a PSC member to apply the "no stale" label.