Per Script Worker Queue Setting

[jchambers2012]

NetBox version

v3.7.8 and v4.0.5

Feature type

New functionality

Proposed functionality

Looking to enhance the script subsystem to add support to specify what worker queue each script should be run in. This will allow critical scripts to run before background or scheduled scripts. This feature would add a new field in the scripts’ Meta possibly called “rq_queue_name” that can bypass the logic in get_queue_for_model and just use the user defined queue. If “rq_queue_name” is not set then the current function can be called to determine what queue the script should be map too.

Use case

We have a few CI/DI pipeline that hit the API to generate control files in a centralized location versus the logic be distributed across different project code. This simplifies management of the project files but these critical scripts must wait in the queue with other background and user run scripts delaying the running of them.

Some of our non-critical script that update data can take 5 minutes to run and they are not time critical like the CI/DI pipeline.

Database changes

I don’t think any models will need to be updated to support this functionality.

External dependencies

n/a

[jchambers2012]

I mocked up what i think needs to be changed but have not tested - let me know if you want me to work on the feature

https://github.com/netbox-community/netbox/compare/develop...jchambers2012:netbox:script_rq_queue_name?expand=1

[arthanson]

@jchambers2012 I've assigned it to you as you said you have some code for it.

[jchambers2012]

Stimulation are working will try and button up the PR this afternoon if not tomorrow

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

[jeremystretch]

While I agree that it would be valuable to enable mapping custom scripts to specific worker queues, we'll need to address a reasonably flexible mechanism for doing so. Statically mapping each script via an attribute (as in PR #16552) is not sufficiently scalable or easy to manage.

[alehaa]

I propose splitting this into two tasks, maybe solving the issue of scalability:

1. Add a queue parameter to Job.enqueue() and the JobRunner framework so that background jobs (including scripts) can define a worker queue to use. This can be any string, usually one of high, default or low. Plugins can also use their own custom queues if needed.
2. Move the queue definition to a ChoiceSet and make it available in the Scripts form. Users would then only be able to select queues defined by NetBox administrators, not those defined by plugins. This should be quite flexible and not overwhelming. If needed, future changes to NetBox could map these defined queue choices to whatever a new scheduler would need.