Multiple Edges not showing in muti_edge_topology.py topology script

[jedelman8]

Hi, I've been playing around with Autonetkit in Python for a few hours now and have one issue that I can't get past:

I was using the multi_edge_topology.py script to try and get a multi-edge design working, but it doesn't print out more than 1 line between two edges in the diagram. I've verified all 10 edges are there (in Python), but only 6 show on the diagram. Is this an error on my side? Have tried setting "bidirectional" to "True" within the script, but that hasn't worked either.

Demo script I'm using: https://github.com/sk2/autonetkit/blob/master/autonetkit/topologies/multi_edge_topology.py

Thanks in advance, Jason

[sk2]

Hi Jason,

Good catch. The problem is that the phy graph is a multi-edge graph, but the newly created isn't. Bidirectional is close, but that allows an edge from a->b as well as b->a. You want multi_edge: g_ospg = anm.add_overlay("ospf", multi_edge=True)

g_phy = anm['phy']

t1 = anm.add_overlay("t1")
t1.add_nodes_from(g_phy)
t1.add_edges_from(g_phy.edges())
print len(g_phy.edges()), len(t1.edges())

gives 10, 6

Adding with multi_edge:

g_phy = anm['phy']
t2 = anm.add_overlay("t2", multi_edge=True)
t2.add_nodes_from(g_phy)
t2.add_edges_from(g_phy.edges())
print len(g_phy.edges()), len(t2.edges())

gives 10, 10

I'll need to find an intuitive way to decide whether to create single or multi-edge graphs, based on the presence of other single/multi-edge graphs. It needs to be intuitive but also predictable: some overlays may require single edge graphs, and we also don't want changing single to multi on an upstream (such as physical) to cascade to unpredictable changes downstream (such as to routing protocol graphs).

[jedelman8]

Very cool. Apparently, I'm still missing something though.

This is what I have and it's still only "publishing" non-multi-edge diagrams.

It's pretty much tweaked from your original script. Any ideas?

    graph = json_graph.node_link_graph(data)
    anm = autonetkit.anm.NetworkModel()

    g_in = anm.add_overlay("input",multi_edge=True)
    g_in._replace_graph(nx.Graph(graph))

    g_phy = anm["phy"]
    g_phy._replace_graph(graph)

    t2 = anm.add_overlay("t2", multi_edge=True)
    t2.add_nodes_from(g_phy)
    t2.add_edges_from(g_phy.edges())

    print len(g_phy.edges()), len(t2.edges())

    autonetkit.update_http(anm)

This prints 10,10 okay, but the diagram output to the web server only displays one link/edge when there should be multiple.

[sk2]

Thanks. I think the problem is the call to nx.Graph which makes it single edge. Trying nx.MultiGraph might fix it.

I'm still working on making the json importer work properly with multi-edge graphs. I'll update once this is fixed.

It's often easier to either import from an external file such as .json or disk, or to use the API directly, such as

import autonetkit
anm = autonetkit.NetworkModel()
g_in = anm.add_overlay("input", multi_edge=True)

nodes = {'r3': (107, 250), 'r5': (380, 290), 
             'r1': (22, 50), 'r2': (377, 9), 'r4': (571, 229)}

for node, pos in nodes.items():
    node = g_in.add_node(node)
    node.device_type = "router"
    node.x, node.y = pos

r1 = g_in.node("r1") 
r2 = g_in.node("r2")    
g_in.add_edge(r1, r2)
g_in.add_edge(r1, r2)

edges = [("r1", "r3")]
g_in.add_edges_from(edges)

# assign interfaces to the edges
g_in.allocate_input_interfaces()

for node in g_in:
    for index, interface in enumerate(node.physical_interfaces()):
        interface.id = "eth%s" % index

for edge in g_in.edges():
    print edge.src, edge.src_int, "to", edge.dst, edge.dst_int

autonetkit.update_http(anm)

[jedelman8]

Hmmm. Both changes still don't work for me. Does the new script you pasted in work and print 2 edges between r1 and r2 for you? Because if it does, that would be interesting :).

Sorry to be a pain. Much appreciated.

And definitely agree, my goal is to use the API directly. My end goal is to do leaf/spine designs for the Data Center (right now). Need multiple links between spines for MLAG/VPC and multiple connections between each TOR back to a core/spine node. Just so you have some more context.

[sk2]

Sorry for the teething issues - we're adding more test cases, but most of the focus has been on the workflow from a specific XML input format :)

One thing that might be going on is using an older version of ANK. You can install a recent version direct from GitHub:

pip install https://github.com/sk2/autonetkit/archive/v0.10.2.zip

does that help fix it?

Sounds an interesting project. What sort of input data/output data were you looking at?

[jedelman8]

Unfortunately, that didn't help....

If your focus is on an XML input format, can you share an example (with multi-edge)? I def don't mind going with what's the most tested :)

As far as input data, likely CDP/LLDP and other characteristics about the devices and then how it's output is all of matter what I can get working. I've spent 2+ days on nx, pgv, and now ank. And in every lib, multi-edge is causing so many issues or just doesn't work. So, we'll see how it goes.

[sk2]

Yeah multi-edge is a bit of a deviation from classic graph theory. It was a bit of work to get it into ANK and working natively and intuitively (or close to working!)

I've just pushed the latest version to the Python Package index.

$ pip install --upgrade autonetkit
$ wget https://raw.githubusercontent.com/sk2/autonetkit/master/autonetkit/topologies/multi_edge.json
$ python test.py
[r4, r5, r1, r2, r3]
[input: (r4, r5, 0), input: (r4, r2, 0), input: (r4, r2, 1), input: (r5, r3, 0), input: (r1, r2, 0), input: (r1, r2, 1), input: (r1, r3, 0), input: (r1, r3, 1), input: (r1, r3, 2), input: (r2, r3, 0)]

and test.py is

import autonetkit
anm = autonetkit.initialize("multi_edge.json")
print anm['input'].nodes()
print anm['input'].edges()
autonetkit.update_http(anm)

I've also updated the workflow step-by-step guide at http://sk2.github.io/autonetkit/tutorial/extending.html to the 0.10 API

Once you get the core API up and running, it should be relatively easy to use the ANK APIs to set/get attributes, and create new overlays based on those attributes.

[jedelman8]

I've never studied graph theory, but this just seems to make sense :)

Four things documented below: output diagram, test.py source code, the output of test.py, and the output during the upgrade process.

Still stumped. I even tried re-installing nx from source before and after the ank upgrade. Tried re-installing the vis server too.

Screen shot attached below

Output running python test.py

cisco@edelman:~/diagrams$ python test.py
[r4, r5, r1, r2, r3]
[input: (r4, r5, 0), input: (r4, r2, 0), input: (r4, r2, 1), input: (r5, r3, 0), input: (r1, r2, 0), input: (r1, r2, 1), input: (r1, r3, 0), input: (r1, r3, 1), input: (r1, r3, 2), input: (r2, r3, 0)]

test.py

#!/usr/bin/env python

def main():

    import autonetkit
    anm = autonetkit.initialize("multi_edge.json")
    print anm['input'].nodes()
    print anm['input'].edges()
    autonetkit.update_http(anm)

if __name__ == "__main__":
    main()

Output upgrading autonetkit

cisco@edelman:~/diagrams$ sudo pip install --upgrade autonetkit
sudo: unable to resolve host edelman
Downloading/unpacking autonetkit
  Downloading autonetkit-0.10.7.tar.gz (114Kb): 114Kb downloaded
  Running setup.py egg_info for package autonetkit

    warning: no files found matching '*' under directory 'autonetkit/textfsm'
Downloading/unpacking netaddr==0.7.10 (from autonetkit)
  Downloading netaddr-0.7.10.tar.gz (1.3Mb): 1.3Mb downloaded
  Running setup.py egg_info for package netaddr

    warning: no previously-included files matching '*.svn*' found anywhere in distribution
    warning: no previously-included files matching '*.git*' found anywhere in distribution
Downloading/unpacking mako==0.8 (from autonetkit)
  Downloading Mako-0.8.0.tar.gz (407Kb): 407Kb downloaded
  Running setup.py egg_info for package mako

    warning: no files found matching '*.xml' under directory 'examples'
    warning: no files found matching '*.mako' under directory 'examples'
    warning: no files found matching 'ez_setup.py'
    no previously-included directories found matching 'doc/build/output'
Downloading/unpacking networkx==1.7 (from autonetkit)
Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 243, in _get_queued_page
    for link in page.rel_links():
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 515, in rel_links
    for url in self.scraped_rel_links():
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 542, in scraped_rel_links
    url = match.group(1) or match.group(2) or match.group(3)
IndexError: no such group

  Downloading networkx-1.7.tar.gz (728Kb): 728Kb downloaded
  Running setup.py egg_info for package networkx

    warning: no files found matching 'scripts/*'
    warning: no files found matching 'networkx/tests/*.txt'
    warning: no files found matching 'networkx/*/*/tests/*.txt'
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.svn' found anywhere in distribution
    no previously-included directories found matching 'doc/build'
    no previously-included directories found matching 'doc/source/reference/generated'
    no previously-included directories found matching 'doc/source/examples'
    no previously-included directories found matching 'doc/source/static/examples'
    no previously-included directories found matching 'doc/source/templates/gallery.html'
Downloading/unpacking configobj==4.7.1 (from autonetkit)
  Downloading configobj-4.7.1.tar.gz
  Running setup.py egg_info for package configobj

Downloading/unpacking tornado>=3.0.1 (from autonetkit)
  Downloading tornado-4.0.2.tar.gz (315Kb): 315Kb downloaded
  Running setup.py egg_info for package tornado

Downloading/unpacking MarkupSafe>=0.9.2 (from mako==0.8->autonetkit)
  Downloading MarkupSafe-0.23.tar.gz
  Running setup.py egg_info for package MarkupSafe

Downloading/unpacking certifi (from tornado>=3.0.1->autonetkit)
  Downloading certifi-14.05.14.tar.gz (168Kb): 168Kb downloaded
  Running setup.py egg_info for package certifi

Downloading/unpacking backports.ssl-match-hostname (from tornado>=3.0.1->autonetkit)
  Downloading backports.ssl_match_hostname-3.4.0.2.tar.gz
  Running setup.py egg_info for package backports.ssl-match-hostname

Installing collected packages: autonetkit, netaddr, mako, networkx, configobj, tornado, MarkupSafe, certifi, backports.ssl-match-hostname
  Found existing installation: autonetkit 0.10.7
    Uninstalling autonetkit:
      Successfully uninstalled autonetkit
  Running setup.py install for autonetkit

    warning: no files found matching '*' under directory 'autonetkit/textfsm'
    Installing autonetkit script to /usr/local/bin
    Installing ank_collect_server script to /usr/local/bin
    Installing ank_webserver script to /usr/local/bin
  Found existing installation: netaddr 0.7.10
    Uninstalling netaddr:
      Successfully uninstalled netaddr
  Running setup.py install for netaddr
    changing mode of build/scripts-2.7/netaddr from 644 to 755

    warning: no previously-included files matching '*.svn*' found anywhere in distribution
    warning: no previously-included files matching '*.git*' found anywhere in distribution
    changing mode of /usr/local/bin/netaddr to 755
  Found existing installation: Mako 0.8.0
    Uninstalling Mako:
      Successfully uninstalled Mako
  Running setup.py install for mako
    changing mode of build/scripts-2.7/mako-render from 644 to 755

    warning: no files found matching '*.xml' under directory 'examples'
    warning: no files found matching '*.mako' under directory 'examples'
    warning: no files found matching 'ez_setup.py'
    no previously-included directories found matching 'doc/build/output'
    changing mode of /usr/local/bin/mako-render to 755
  Found existing installation: networkx 1.7
    Uninstalling networkx:
      Successfully uninstalled networkx
  Running setup.py install for networkx
    deleting networkx.egg-info/requires.txt

    warning: no files found matching 'scripts/*'
    warning: no files found matching 'networkx/tests/*.txt'
    warning: no files found matching 'networkx/*/*/tests/*.txt'
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '.svn' found anywhere in distribution
    no previously-included directories found matching 'doc/build'
    no previously-included directories found matching 'doc/source/reference/generated'
    no previously-included directories found matching 'doc/source/examples'
    no previously-included directories found matching 'doc/source/static/examples'
    no previously-included directories found matching 'doc/source/templates/gallery.html'
  Found existing installation: configobj 4.7.1
    Uninstalling configobj:
      Successfully uninstalled configobj
  Running setup.py install for configobj

  Found existing installation: tornado 4.0.2
    Uninstalling tornado:
      Successfully uninstalled tornado
  Running setup.py install for tornado
    building 'tornado.speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c tornado/speedups.c -o build/temp.linux-x86_64-2.7/tornado/speedups.o
    tornado/speedups.c:49:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/tornado/speedups.o -o build/lib.linux-x86_64-2.7/tornado/speedups.so

  Found existing installation: MarkupSafe 0.23
    Uninstalling MarkupSafe:
      Successfully uninstalled MarkupSafe
  Running setup.py install for MarkupSafe

    building 'markupsafe._speedups' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o build/lib.linux-x86_64-2.7/markupsafe/_speedups.so
  Found existing installation: certifi 14.05.14
    Uninstalling certifi:
      Successfully uninstalled certifi
  Running setup.py install for certifi

  Found existing installation: backports.ssl-match-hostname 3.4.0.2
    Uninstalling backports.ssl-match-hostname:
      Successfully uninstalled backports.ssl-match-hostname
  Running setup.py install for backports.ssl-match-hostname

Successfully installed autonetkit netaddr mako networkx configobj tornado MarkupSafe certifi backports.ssl-match-hostname
Cleaning up...

[sk2]

Hi Jason,

It looks like the core code is doing the right thing - you can see multiple links from the third item in the tuple:

[input: (r4, r5, 0), input: (r4, r2, 0), input: (r4, r2, 1), input: (r5, r3, 0), input: (r1, r2, 0), input: (r1, r2, 1), input: (r1, r3, 0), input: (r1, r3, 1), input: (r1, r3, 2), input: (r2, r3, 0)]

where the "input" is the overlay, and then the tuple is (src, dst, key) We can see two links from r4 to r2, and 2 links from r1 to r2, and three links from r1 to r3.

The only think I can think of is that you are running an older version of the visualisation package. Can you try upgrading it:

pip install --upgrade autonetkit_vis

and then restart the visualisation webserver.

I just did a clean install inside a Python virtualenv, with autonetkit_vis-0.1.7 and it looks correct on the visualisation.

My testing:

$ virtualenv ank10
$ cd ank10/
$ source bin/activate
$ pip install autonetkit
$ wget https://raw.githubusercontent.com/sk2/autonetkit/master/autonetkit/topologies/multi_edge.json
$ wget https://gist.githubusercontent.com/sk2/380ff02430f6733aa727/raw/013a8b43d9e74f7a562a00f196819a729a862cc5/gistfile1.txt
$ mv gistfile1.txt test.py
$ python test.py

gives

[r4, r5, r1, r2, r3]
[input: (r4, r5, 0), input: (r4, r2, 0), input: (r4, r2, 1), input: (r5, r3, 0), input: (r1, r2, 0), input: (r1, r2, 1), input: (r1, r3, 0), input: (r1, r3, 1), input: (r1, r3, 2), input: (r2, r3, 0)]

and the visualisation server

$ pip install autonetkit_vis
$ ank_webserver 

Some of the graph theory is very powerful, such as neighbors, paths, connected components, and the merge/split functions. I've tried to distill the core concepts down to make them intuitive. Some were mentioned in the AutoNetkit presentations, and there's ongoing work to document the API further.

Also - if you're doing prototyping ipython notebook is well worth checking out - much easier to play around with than writing Python scripts. I usually use the notebook when designing a new feature, and then once it's working I move it into a .py script.

[jedelman8]

Just did a clean install (on a box that didn't have any graph applications) and still no dice. I wonder if there are other dependencies. FWIW, I'm trying on Ubuntu 12.04/14.04 running Python 2.7.

Thanks for the help. Glad to know calculations are correct although the diagram is what I really wanted :)

I may have to throw in the towel if I don't get this by tomorrow though. If you can think of anything else, feel free to let me know.

[sk2]

That's odd. What does

pip freeze |grep autonetkit

say? It should be

autonetkit==0.10.7
autonetkit-vis==0.1.7

Sorry this has been so painful. I think there's something with the pip dependencies.

I pinged you on skype (assuming same username) if that's easier to 1:1 to try and work it out.

Not sure of any other packages in this area for working with multiple edge graphs, other than networkx.

[jedelman8]

I'll sign in now. Thank you.

spkadmin@SPK-dist-A01:~$ pip freeze |grep autonetkit autonetkit==0.10.7 autonetkit-vis==0.1.7 spkadmin@SPK-dist-A01:~$

On Tue, Nov 4, 2014 at 7:20 PM, Simon Knight notifications@github.com wrote:

That's odd. What does

pip freeze |grep autonetkit

say? It should be

autonetkit==0.10.7 autonetkit-vis==0.1.7

Sorry this has been so painful. I think there's something with the pip dependencies.

I pinged you on skype (assuming same username) if that's easier to 1:1 to try and work it out.

Not sure of any other packages in this area for working with multiple edge graphs, other than networkx.

— Reply to this email directly or view it on GitHub https://github.com/sk2/autonetkit/issues/254#issuecomment-61739887.

[jedelman8]

Signing off Skype (jedelman8) now for about 2-2.5 hours. Maybe we can chat later...if not, tomorrow. Would LOVE to get this working.

[sk2]

I've just pushed an updated version of the visualisation script. I think there was an issue in the javascript - sorry!

0.1.8 should work correctly. Just to be sure:

$ pip uninstall autonetkit_vis
$ $ pip install autonetkit_vis
Downloading/unpacking autonetkit-vis
  Downloading autonetkit_vis-0.1.8.tar.gz (427kB): 427kB downloaded

Hopefully that fixes it.

[jedelman8]

Cool....I think we have a winner....maybe I can start with "real" questions now :)

Thanks so much. I've attached a screen shot b/c I still think there is a small bug now in the drop down menus. Notice how you can't read the drop down options. Note: you can read them once you click any of the drop downs.

I may ping you on your gmail address if anything else arises if that's easier. Pretty sure I saw it in some of the source code.

[sk2]

Nice work. Sorry about the installation problems. Once it's up and running it should be a bit smoother - the core API has been used and tested pretty extensively.

I'll have a look at the drop-down, I think it's my CSS in Firefox getting a bit out.

Yep feel free to ping me on gmail - firstname.lastname at gmail.com I'm interested in a real-world use case of the API :)