JS based site topology map

[bashioo]

Issue type

[x] Feature request [ ] Bug report [ ] Documentation

Description

Hi, I've created per site topology maps for netbox using vis.js library http://visjs.org (licensed under both Apache 2.0 and MIT). :

Here is a live demo: http://114.142.160.109:8000/dcim/sites/ , check sites' pages the site hosts in Australia so it might feel slow for the rest of the world. use testuser/testpassword to get write access to sites, devices and connections. Connections can be created and deleted via Edit Site page. Feel free to try it. Edit interface:

Here is the fork: https://github.com/bashioo/netbox/tree/topology-2.3 I will keep it in sync with the preview site above, you can use it if you want to try it on your server. Installation is the same as for the original netbox.

THIS IS STILL AN EARLY DEVELOPMENT. Please do not use it in production.

The addon draws connections between devices. Device picture is determined by Device role. By default it uses visjs solver to position devices, once coordinates are saved (via edit site page) layout is fixed. It should also be possible to extend functionality to allow add/modify/delete interfaces, Example: http://visjs.org/examples/network/other/manipulationEditEdgeNoDrag.html

I had to modify Device model to add "coordinates" field accessible via API only to store device coordinates.

Is it something you would be interested in adding to Netbox? It is sure raw now and needs some work but it looks nice and helps visualise sites topology way better than the current topology maps.

Regards, Vas

[jvanderaa]

Something that I know I'd like to see on a future topology type tool would be if you could also do check boxes for including interface names and networks & IP addresses (Network in the middle, and last octet on the device). Just some ideas to continue to possibly enhance this, as I personally like what I see.

[larsuhartmann]

Wow! This looks pretty amazing for a first look!!! It would be super nice to have this in a future netbox release!

Really Nice work you did there!

[larsuhartmann]

Would it be possible to build visjs into the existing topology solution? As in let visjs render the selected devices into "nicer" topology views?

[bashioo]

@vanderaaj, I guess it is possible to add any information we have in DB but for prefixes and IPs it might be challenging though as they are logical and not always directly correlate to physical interfaces. We will have to do lots of API calls to get all required information (two current requests, then: get interfaces by device => for each interface's IP get parent prefix => compare prefixes on both ends (for situations like when an L3 interface on a router is connected to L2 switch), etc). With that amount of details it is probably better to change logic from API calls to separate view, to get all the required information from DB on the server side and then somehow pass it to JS to build the network.

@larsuhartmann, it should be possible. Not sure how the current (old) topology is created but visjs can create hierarchical networks (http://visjs.org/examples/network/layout/hierarchicalLayoutUserdefined.html). We can use the same regexes to figure out which device should go to which level. I've tried to play with hierarchical layout though but found it not as informative/flexible as the one I am using now.

[phurrelmann]

Awesome feature! I also like the distinction between dynamic and static placement for the generated topology

[rdegez]

The map looks awesome! I would love to see this in netbox :-)

[bashioo]

I've made some progress on the topology and now you can toggle static/dynamic layout and most importantly add and delete connections via GUI.

Go to "Edit site" page to modify connections. To add a connection click "Add connection" button and then drag the link between two devices. To delete select any connection first and then click "Delete selected".

Changes are saved to DB immediately via API.

Use testuser/testpassword to get access.

It is very raw at the moment, probably has some bugs but you can get an idea.

[phurrelmann]

Perfect! Just wondering how this would scale with lots of devices on a site.

[bashioo]

Should be OK, tested with 100+ devices without any issues.

[GitRost]

Looks cool

[LBegnaud]

I don't see it said explicitly, but does this new module use the existing interface connections or does it track relationships separately?

Seems like respecting device connections based on existing netbox data would be most useful and perhaps a way to exclude certain things if needed

Edit: also i should mention that this is amazing

[bashioo]

It uses only netbox data of course. For it to work your netbox devices should have interfaces and connections between them (or only interfaces, you can create connections later using this module). To draw the topology its making exactly two API calls: GET /api/dcim/devices/?limit=0&site=SITE - to create all the nodes GET /api/dcim/interface-connections/?limit=0&site=SITE - to create all the connections between nodes

Then when you are adding a new connection it is calling GET /api/dcim/interfaces/?device_id=edgeData for both A and B devices to create dropdowns to select interfaces from. Once interfaces are selected and confirmation button is clicked it is sending POST /api/dcim/interface-connections/ request to actually add the a connection to the DB and on successful reply from the server it adds the edge to the topology diagram. The new edge is created with the proper ID matching the one returned in POST request so it can be immediately deleted by id if required.

On deletion it is calling GET /api/dcim/interface-connections/'+topology.getSelectedEdges()[0]+'/' once to get information about the connection - from, to, interfaces to ask for confirmation. Once delete command is confirmed is is calling DELETE on the same connection and again only on successful reply from the server it deletes the edge for the topology.

[slindermann]

Great work, thank you for this! Would you be able to supply a short beginners Howto for implementing this in a master branch netbox setup? (Still testing though, non productive)

[bdlamprecht]

This is great work, I know that I found the current way of creating Topology Maps very simplistic (although not in a bad way). But this improves on it at least 10 fold. I know I would definitely like to see this integrated in a future NetBox release.

[bashioo]

@slindermann you can install it in the same way you install original netbox, just use my github fork from the first message instead of the original netbox, use branch name visjs_topoogy, I've just pushed all the current changes there.

Do not use it with your production DB though, as it will modify DB to add "coordinates" field to "device" model. When updating testing installation to this one do not forget to run upgrade.sh to create a new field in the DB.

I've also created a test user for the live demo site above. Please check the first message for details.

[bashioo]

I've now created a separate branch based on develop-2.3 of the Netbox. It works pretty much the same just needed to modify some logic around Virtual Chassis support. https://github.com/bashioo/netbox/tree/topology-2.3 The demo site is using 2.3 based branch as well now.

@jeremystretch should I keep working on it the way it is now? Are you OK with using API calls to draw and save changes, device model change, site pages layout changes etc?

Next steps for me would be to optimise code, add connection statuses, maybe add color coding for different connections (ethernet vs serial vs stack), add error handling, add some beauty to dialogs and finally prepare a PR ( this one would require a lot of googling as I am not a programmer at all).

Please let me know what are you thinking.

Cheers

[bashioo]

Alrighty, I have not heard anything back from @jeremystretch so he either missed it or he is cool with it =)

I'm now at the stage where I am almost happy with the result and I am asking those willing to participate to help me to polish the stuff before I create a pull request.

The test site is still there, you can get limited access using credentials from the first message. Branch at https://github.com/bashioo/netbox/tree/topology-2.3 is updated just now with the most recent code.

If you want to help with the module, please check the test site, check source code (it might be bad, I am not a programmer), you suggestions are welcome. If you could contribute with some cool device icons that would be awesome too.

After some testing I will submit a PR probably in about a week.

Cheers

[jeremystretch]

While this is impressive work, unfortunately I have to reject this for several reasons.

1. We're not currently accepting major new FRs due to the backlog of existing FRs (many of which have been open for over a year).

2. This would essentially extend NetBox to become a drawing tool, which is well beyond its intended scope. (The topology maps feature which exists today is intentionally limited and requires minimal overhead.)

3. Implementing this FR would beget an entire new tract of bug reports and feature requests around this functionality, further slowing a development pace which I'm already struggling to maintain.

4. There's no reason this functionality can't exist as a standalone tool, integrating with NetBox through its REST API.

Again, it's impressive work for sure, and I wish you had raised the idea here first before sinking so much time into it. Unfortunately, it's not something we can add to NetBox in the foreseeable future, but I encourage you to continue developing it as a standalone extension.

[bashioo]

Thanks for the review, I kinda actually expected something like this.

The decision is yours, sure, I just don't really agree with your reasoning, because:

1. This one would not add any overhead to the current FR backlog as it is already implemented.
2. Agreed, with editing I've went too far, it is just something I was really interested to try to implement. There are no issues with limiting functionality to the exactly same level the current topology view provides, just looking better. The main idea here was to visualise, not to provide drawing functionality.
3. As per the previous point, limiting functionality to view only would simplify the code dramatically, just two API calls to draw everything.
4. There are, actually. Trying to duplicate Netbox site choice, maintaining authentication (in case of full functionality), dealing with cross site AJAX, publishing (apache virtual host? just for a JS generated picture? overkill IMO). I've created this for my company as I am moving all devices' information from local wiki into Netbox and topology in the same place as all the rest information seemed logical (well, Netbox has a topology, but once I tried it I realised it is no fit for my goal). Having it separate, well, I'd better just leave it in the wiki rather than creating a third place to store part of the information at.

And I've raised this idea here well before (like 17 days ago) but never got a reply, so I just wanted to try to implement it (I like challenges, yes =)). If there is no chance for it to become a part of the official Netbox so be it, I'll just leave it as a fork.

Cheers

[slindermann]

@bashioo I can only encourage you to keep the good work up and I'm happy to stick with your fork as long as I'll be able to apply updates and patches from the original netbox git. Seems like I need to work through the github docs a little more to understand how this works ;)

I personally think this is a very usefull feature and crucial site information.

[cimnine]

@bashioo My suggestion is that you maintain your feature not as a fork, but as an individual module (if possible). My team's currently working on such a module (for GraphQL support, not ready for prime-time (yet)!) as well, which we maintain as independent Python/Django module.

The advantage for users is that it's relatively easy to add multiple 3rd party functionalities bundled in modules to their netbox installation. But if everyone maintains their fork, this is not so simple.

[cimnine]

And maybe Jeremy is even willing to implement certain adjustments so that such 3rd party modules can easily integrate (e.g. into the navigation) if this would become necessary. (As I am certain this will not be the last feature that is not going to be merged into and maintained as part of Netbox, but in which many users are interested regardless.)

[slindermann]

+1 on the last suggestion. It would be neat to add this as an optional module and be able to stick with the original branches.

[larsuhartmann]

+1 from me too - it would be a shame to have this running as a fork - for me this feature is too nice to miss out on!

It would be nice to have some sort of plugin-system - as in my opinion the alternative would be to end up with forks and all the resulting merge-hell...

[bashioo]

Thanks for your support guys! I would definitely keep it running, as I created it for my company in the first place and then decided to contribute to netbox. As for the way to implement it, there are multiple options but I really only see it as a part of the site view page (but I am open to suggestions), therefore a separate Django app would not be able to satisfy this requirement (it will have to use a separate url). Running a fork is too much overhead either, as I want to keep the system up to date with the official releases. So the only viable option I can think of is some kind of small package with bash/python script to patch the existing Netbox installation. I'll have to create one for every version of netbox and it would simply install required JS and CSS files to proper destinations and patch site.html template. For data model I will probably have to revert back to using a custom field to store coordinates as I do not really want to modify a production DB schema this way. So the installation process would be as simple as downloading the archive, unpacking and running install.sh or similar. Then the admin will have to create a custom field using supplied guide (only for the first installation, would not be required for updates) and restart supervisord/netbox.

[lampwins]

I have not really been following along but from what I can see, this is mostly front end, excluding the coordinates part.

Have you thought about a chrome (or other browser) plugin? People have been talking about a "plugin" architecture for netbox lately and for front end stuff, this would almost certainly fit the bill. The thing people need to keep in mind, is that supporting user (or otherwise one off modification) is a huge undertaking from a basic architectural standpoint.

Given the use case here, I think a browser plugin would work but you would have to figure out the coordinates portion.

[dgarros]

it would be very useful to have a way to add element(s) to the navigation bar and a plugin infrastructure. we are just starting to use Netbox in my organization but we already have some needs that would require our own module(s)/plugin(s). Not sure yet how to do that in a maintainable way ...

[bashioo]

I've started a new discussion here: https://groups.google.com/forum/#!topic/netbox-discuss/MRp0I6nS5xc It is probably better to move there instead of sitting in this closed issue. Added some new stuff: circuit terminations are now can be displayed as well.

[candlerb]

Worth mentioning drawthe.net (demo) as possible library to use