search

eNMS-automation / eNMS

An enterprise-grade vendor-agnostic network automation platform.
https://www.enms.io/
GNU General Public License v3.0
809 stars 158 forks source link

netbox support #19

Closed afourmy closed 5 years ago

afourmy commented 6 years ago
OzNetNerd commented 6 years ago

What data would you want stored in the eNMS DB? Rather than duplicate data across two DBs, would it be more appropriate to query the netbox API as required?

afourmy commented 6 years ago

I've never used netbox (I was planning on installing it to get started with this issue), so I don't know for sure what data netbox stores in its database (and more importantly, what data can be retrieved via netbox REST API).

I would like to the retrieve all nodes and links in netbox. In eNMS, there are several types of nodes and links as you can see here: https://github.com/afourmy/eNMS/blob/master/source/objects/models.py. As far as I know, netbox does not have any visualization feature, but in eNMS, each type of node has it's own icon when displayed in the geographical and logical views, and each type of link it's own color so having the whole topology makes sense to propose a visualization of the network. We can imagine to create new types of nodes/links to better fit netbox model if needed. As for the property, eNMS would need at the very least the IP of the device, but if there are other properties that we think make sense importing, we can update the object models in eNMS to import them too.

It was suggested in https://github.com/afourmy/eNMS/issues/23 to pull the user settings from netbox. Not sure netbox REST API allows that, but if it does, pulling the list of all netbox users (and their privileges / access rights if possible) would make sense. We also need the network credentials, but I doubt it's possible to retrieve that from netbox.

There are currently no ports / interfaces in eNMS, but I created https://github.com/afourmy/eNMS/issues/30 to add support for that. If we do (that's a very big change though), we can import netbox data about ports and interfaces (IP, etc).... but we are in "netbox territory" here 😄

OzNetNerd commented 6 years ago

Unfortunately I haven't had a chance to play around with it yet either. In regards to what can be retrieved via the API, the docs say:

circuits, DCIM, extras, IPAM, secrets, and tenancy

Speaking of which, we can use pynetbox to interact with it.

In regards netbox to visualization, there's talk of it here, here and here. Though as I'm yet to use it, I'm not sure if there's overlap between it and eNMS.

It was suggested in #23 to pull the user settings from netbox. Not sure netbox REST API allows that, but if it does, pulling the list of all netbox users (and their privileges / access rights if possible) would make sense. We also need the network credentials, but I doubt it's possible to retrieve that from netbox.

Perhaps netbox's use of LDAP is the way to go? That way the solution will work regardless of whether people use netbox.

There are currently no ports / interfaces in eNMS, but I created #30 to add support for that. If we do (that's a very big change though), we can import netbox data about ports and interfaces (IP, etc).... but we are in "netbox territory" here 😄

Yes, we're going to have to make sure we don't re-invent the wheel by overlapping too much with what netbox has already done 😆

By the way, a couple of other bits you might be interested in: Integrate NAPALM to support pulling live data from devices #1348 and NAPALM Automation

afourmy commented 6 years ago

I had a look at pynetbox, it seems very limited. Since eNMS is primarily focused on network automation, I think that what we should pull from netbox first and foremost is the list of all devices, along with their IP addresses (and ideally their GPS coordinates for the geographical view to work)

In regards netbox to visualization, there's talk of it here, here and here. Though as I'm yet to use it, I'm not sure if there's overlap between it and eNMS.

I cannot find any screenshot or demo of what they call "Topology map". I think it might just be a static view of the topology filtered with regular expressions.

Perhaps netbox's use of LDAP is the way to go? That way the solution will work regardless of whether people use netbox.

eNMS can already authenticate users by configuring a TACACS+ server (http://afourmy.pythonanywhere.com/admin/administration), and there is an open issue for RADIUS in which LDAP was mentioned (https://github.com/afourmy/eNMS/issues/32). I believe the code for RADIUS and LDAP would be similar to what's already implemented for TACACS+.

Yes, we're going to have to make sure we don't re-invent the wheel by overlapping too much with what netbox has already done

Yes, as I see it, the inventory / visualization features in eNMS are mainly here to ease the network automation process.

By the way, a couple of other bits you might be interested in: Integrate NAPALM to support pulling live data from devices #1348 and NAPALM Automation

Thanks for the links :) At the last NAPALM hackathon, I remember there was a demo by the creator of netbox about this. AFAIK, it uses NAPALM to pull some data about all interfaces, and report any inconsistency between netbox (nsot) and the network.

OfWolfAndMan commented 6 years ago

@afourmy I have looked at Netbox a couple of times, and I think it could be a great option for adding data even as an alternative dynamic discovery mechanism.

What I see, based on fields and explicitly speaking in the context of devices only, is you could get everything with a REST call, with the exception of the lat/long and the OS version. However, I think having a background script that combines NTC's ntc-templates with a simple "show version" command could do the trick (Use the data from the REST call to grab the management IP). The latitude/longitude is a bit more tricky. I'd have to think about that one...

afourmy commented 6 years ago

I think there are at least 3 distinct topics here (and probably 2 github issues) that can be addressed separately:

OfWolfAndMan commented 6 years ago

@afourmy I was speaking of OS version, not the OS type, although I am glad you showed me that class, as I may consider using it elsewhere. It appears we could use the .1.3.6.1.4.1.9.9.25.1.1.1.2.7 MIB, but I'm sure there are other options.

Network topology from Netbox is a definite.

As for lat/long, yes, any ISP database should have this information, and I know that eNMS is targeted at an ISP, but this tool could be useful to Enterprise environments as well. Either they would omit the information and fill it with a N/A or Null value, or a separate view of the UI could be formed. I'm sure this would take some work, but it's an idea (One that should be pursued later, if necessary, considering its minor priority relative to the aforementioned goals).

afourmy commented 6 years ago

About the OS version, I don't think it's a problem if we cannot retrieve it from netbox, because eNMS focuses on network automation and in order to use netmiko or napalm, the only thing we really need is the OS type (+ IP + credentials).

Now that the idea of "Custom scripts" is implemented in eNMS (https://enms.readthedocs.io/en/latest/automation/scripts.html#custom-script) and with netmiko recent support for ntc templates (https://pynet.twb-tech.com/blog/automation/netmiko-textfsm.html), what we can do is create a custom script that uses netmiko and a NTC template to update the OS version of all nodes.

Regarding the lat/lon, it is not mandatory right now for eNMS to work. In the latest official release, you can use the logical view based on d3.js to display the network topology with a force-based algorithm (along with filters if needed), and in the latest version of eNMS in the master branch, there is a new mechanism (yet undocumented) requested by esh in the slack channel to send/schedule a script/workflow directly from the script/workflow management table (i.e you no longer need to use the views to send a script). There is a new Run button in these tables for that.

afourmy commented 5 years ago

This is done. I use pynetbox to import all devices from netbox, as well as their IP address (IPv4 or IPv6). Netbox does not store any GIS data such as longitude and latitude for now, so that part cannot be done.