support multiple workers to accommodate different versions

FlorianHeigl commented 3 years ago

Desired Behavior

Be able to support devices that have contradicting pip dedepencies. We can built two containers, for example netbox-worker-legacy and netbox-worker.

Contrast to Current Behavior

currently you might need to upgrade netmiko/napalm for some modules

# hp procurve etc. (there are >3 plugins but this one seems easiest)
RUN pip3 install napalm-procurve
# cisco SG series
RUN pip3 install napalm_s350
# qrcode generator
RUN pip3 install netbox-qrcode
# autodiscovery (not recommended)
RUN pip3 install np-autodiscovery

some others will not run because they need older versions and the dependencies will clash some examples of real hardware that would be supported, but adding its support becomes impossible because of the above.

# huawei vrp
RUN pip3 install napalm-huawei-vrp
# mellanox
RUN cd napalm-onyx-master && pip3 install -r requirements.txt
# alcatel-lucent omniswitch series (6860 etc)
RUN pip3 install napalm-aos

Changes Required

Be able to run multiple netbox-worker instances, that have different versions of napalm and netmiko installed. (i do not know how to set this on the netbox side, but since a worker is supported in general, i hope that the groundworks already exist since that was introducted)

Discussion: Benefits and Drawbacks

The truth is that many of the napalm modules don't see enough maintenance to stay up to date, and just as such they do not stay in sync with netbox. then there's also the case of napalm modules that disappeared in recent napalm versions (i.e. avaya/nortel) In result, there's always only a fraction of functioning modules, and the worst part is that what is is usable from within netbox is the lowest common denominator. This is (imo) unneccessary, the abstraction through napalm is good enough to desire a wide support.

what i suggest is probably the easiest path to solve this. a worker could be chosen based on the platforms defined.

Alternatives approaches

there's probably also other ways using virtualenv, but i have no understanding of that or if it would fit
solving it only upstream by forward-porting: there's multiple drivers where it's unrealistic to wait for an upstream solution, since they already sit on open PRs for months/years
solving it only upstream by forward-porting: the overall ecosystems don't (yet) stay in sync. i am not sure if there's i.e. tests for netbox-docker that check if things from netbox-community work at a given point in time

tobiasge commented 3 years ago

That is not something we can solve with multiple workers. The workers in Netbox are only used to process webhooks, as far as I can see. The NAPALM request are done from the thread and therefore image where the web requests are processed. For the functionality you describe Netbox had to support to run NAPALM request in the worker (and there in a specific queue). You should open an FR an the main Netbox repository.

What you could do as a workaround: Run multiple Netbox images an configure them to use the same database. Then you could start your NAPLAM request in the corresponding image which you select by the URL you use.

FlorianHeigl commented 3 years ago

@tobiasge Thanks for the explanation. I'm gonna close the issue since it's out of scope. Gonna try if I can make the workaround fly, too ;-) I'll put a comment when I make the upstream feature request.

netbox-community / netbox-docker