Closed alehaa closed 1 month ago
I think it would be very useful if netbox support easier usage of the jobs for plugins. Im actually trying to implement a job for a plugin, it's very tricky to find how it works.
A lot of plugins implement a custom solution with the big disadvantage that the jobs not listed in the Jobs Tab.
@alehaa thanks for the detailed FR and example code! We've selected this as a milestone initiative for NetBox v4.1. I was set to work on it myself, but would you like to contribute a PR for it?
@jeremystretch sure I can work on it.
@alehaa great! Please be sure to base your PR off of the feature
branch as this will be going into NetBox v4.1.
@alehaa I have done similar things in my plugins to achieve similar results. If you want we can collaborate on this feature in your fork. Just let me know. I'm also on slack with the same username if you want to reach out.
Thanks for your time :)
@alehaa can you share what progress you've made on this?
@jeremystretch Unfortunately I didn't start yet but plan on submitting a PR this week.
Despite the missing documentation pages, the BackgroundJob
implementation is ready for merge. The jobs for synchronizing data sources and running scripts (background and interactive) have also been migrated to this new feature.
However, for adding SystemJob
I encountered a problem: While my implementation documents to use the plugin's ready
function, Django discourages this behavior because the database may not be ready. @jsenecal have you encountered a similar problem and can share some ideas to solve this? Otherwise I could share my current efforts in a PR and submit a second one for background system jobs at a later time @jeremystretch?
@alehaa please submit what you have for now as a draft PR and I'll take a look. Thanks!
@jeremystretch I've submitted a new draft PR related to this issue.
Additional ScheduledJob
and SystemJob
implementations can be found in the original issue body under item 4. However, I haven't committed these because I haven't found a way to use them in practice yet. I didn't want to commit something that could not be used.
After giving it some thought, I see no built-in option for this in Django and some custom logic will need to be used to solve this. I could imagine sending a signal just before the WSGI application accepts requests to set up background jobs. ScheduledJob
and SystemJob
will then need to register for this signal to be scheduled when needed. In addition, a management command can be used to manually schedule all registered background jobs interactively. This way, the registration is done only once, instead of every time a ready
function is executed.
Would this be acceptable and in line with the general NetBox codebase?
I've found a solution that works without introducing new methods or custom signals by reusing Django's connection_created
signal. This allows the database to be accessed on application startup, but without any interaction. This signal will be issued after all plugins have been loaded so no warning is issued in the logs. I'll continue working on my PR and submit results ASAP.
NetBox version
v3.7.5
Feature type
Change to existing functionality
Proposed functionality
NetBox already includes the functionality to schedule jobs using either the
django-rq
library or thecore.Job
model combined with theJobsMixin
class. Making the abstract job functionality publicly available to plugins allows decoupling from the backend ofdjango-rq
and a consistent experience in NetBox across all functionality. For this I propose:Add
netbox.models.JobsMixin
to the list of API available model mixins in the documentation. This allows plugins to implement new models with Jobs enabled.Add a new
BackgroundJob
class to implement the execution of the job's code, i.e. for consistency in callingstart
,terminate
and setting error messages. This class should also be used in existing NetBox functionality to run background jobs for consistency. Below is a sample implementation from a plugin of mine for demonstration purposes.Optional: Enhance the
core.models.Job.enqueue()
method with aqueue
parameter to schedule jobs for specific queues (e.g.low
,high
). If not provided the default queue will be used, so there's no change in API compatibility.Optional: Add the ability to schedule system background tasks by plugins, e.g. for periodic synchronization with other systems. Below is a sample implementation from a plugin of mine for demonstration purposes.
Use case
Plugins get a standardized interface for adding models with jobs enabled using the
JobsMixin
, just like native NetBox models. This provides a consistent experience.The environment for running background jobs will be standardized, as startup, termination, and error handling will be the same for all jobs. Individual jobs don't have to worry about rescheduling, but can rely on well-tested and managed code.
Using the
SystemJob
interface, plugins could schedule system tasks such as periodic synchronization with other systems (e.g. virtualization clusters) or perform housekeeping. These jobs are usually not bound to a specific NetBox object and currently require either direct access to thedjango-rq
library or use of an external cronjob and management commands.Database changes
None
External dependencies
None
For the functionality described above I can share my existing code, add test cases and provide a PR for review. Special thanks goes to @wouterdebruijn for sharing his ideas and feedback in the NetBox discussions.