search

codeout / inet-henge

Generate d3.js based Network Diagram from JSON data.
MIT License
258 stars 38 forks source link

Possible solution for hierarchical design #27

Open marhyno opened 3 years ago

marhyno commented 3 years ago

Hi, after last update I would like to point out one possible solution for hierarchical design but I would need your help. I found really good tutorial here: https://www.alyne.com/en/blog/tech-talk/using-webcola-and-d3-js-to-create-hierarchical-layout/

Here are the steps I took: 1.Changed diagram.ts *a) import as _ from 'lodash'; b)**

initCola(): any { // eslint-disable-line @typescript-eslint/no-explicit-any return cola.d3adaptor() .avoidOverlaps(true) .handleDisconnected(false) .symmetricDiffLinkLengths(100) .linkDistance(100) .size([this.options.width, this.options.height]); }

c) inside render function I added this `const levelGroups = .groupBy(nodes, "level"); console.log(levelGroups)
for (const level of Object.keys(levelGroups)) { const nodeGroup = levelGroups[level]; const constraint = { type: "alignment", axis: "y", offsets: [], }; let prevNodeId = "none"; for (const node of nodeGroup) { constraint.offsets.push({ node: .findIndex(nodes, (d) => d.name === node.name), offset: 0, });

              if (prevNodeId !== "none") {
                console.log(_.findIndex(nodes, (d) => d.name === prevNodeId));
                console.log(_.findIndex(nodes, (d) => d.name === node.name));

                constraints.push({
                  axis: "x",
                  left: _.findIndex(nodes, (d) => d.name === prevNodeId),
                  right:_.findIndex(nodes, (d) => d.name === node.name),
                  gap: 50,
                });
              }

              prevNodeId = node.name;
              console.log(prevNodeId);
            }

            constraints.push(constraint);
        }

this.cola.nodes(nodes)
            .links(links)
            **.constraints(constraints)**
            .groups(groups);
        this.setDistance(this.cola);`

d) in node.ts I added level property in NodeDataType - level: number, in Node Class - public level: number; in constructor - this.level = data.level;

e) in package.json in dependencies I added "@types/lodash": "^4.14.168", "lodash": "^4.17.20",

JSON here - you can see I added level property: { "nodes": [ { "level":1, "name": "testPERMANENT-10.0.1.1", "meta": { "tenant-ip": "10.0.1.1" }, "icon": "/static/images/router.png" }, { "level":1, "name": "testPERMANENT-10.0.1.2", "meta": { "tenant-ip": "10.0.1.2" }, "icon": "/static/images/router.png" }, { "level":2, "name": "INTERNET", "icon": "/static/images/router.png" }, { "level":3, "name": "AZ1-testpermanent-123-muc-vpn1", "meta": { "name": "vpn1" }, "icon": "/static/images/router.png" }, { "level":3, "name": "AZ2-testpermanent-123-muc-vpn2", "meta": { "name": "vpn2" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ1-testpermanent-muc-rs1", "meta": { "name": "rs1" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ1-testpermanent-muc-rs2", "meta": { "name": "rs2" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ2-testpermanent-muc-rs3", "meta": { "name": "rs3" }, "icon": "/static/images/router.png" }, { "level":4, "name": "AZ2-testpermanent-muc-rs4", "meta": { "name": "rs4" }, "icon": "/static/images/router.png" }, { "level":5, "name": "10.0.0.0/24" }, { "level":5, "name": "10.0.1.0/24" }, { "level":6, "name": "POD1-FCIPOCLEAF101", "meta": { "name": "FCIPOCLEAF101" }, "icon": "/static/images/router.png" }, { "level":7, "name": "POD1-FCIPOCLEAF102", "meta": { "name": "FCIPOCLEAF102" }, "icon": "/static/images/router.png" }, { "level":6, "name": "POD2-FCIPOCLEAF202", "meta": { "name": "FCIPOCLEAF202" }, "icon": "/static/images/router.png" }, { "level":7, "name": "POD2-FCIPOCLEAF201", "meta": { "name": "FCIPOCLEAF201" }, "icon": "/static/images/router.png" } ], "links": [ { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-muc-rs1" }, { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-muc-rs2" }, { "source": "10.0.0.0/24", "target": "POD1-FCIPOCLEAF102" }, { "source": "10.0.0.0/24", "target": "POD1-FCIPOCLEAF101" }, { "source": "10.0.0.0/24", "target": "POD2-FCIPOCLEAF201" }, { "source": "10.0.0.0/24", "target": "POD2-FCIPOCLEAF202" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-muc-rs3" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-muc-rs4" }, { "source": "10.0.1.0/24", "target": "POD1-FCIPOCLEAF102" }, { "source": "10.0.1.0/24", "target": "POD1-FCIPOCLEAF101" }, { "source": "10.0.1.0/24", "target": "POD2-FCIPOCLEAF201" }, { "source": "10.0.1.0/24", "target": "POD2-FCIPOCLEAF202" }, { "source": "INTERNET", "target": "AZ1-testpermanent-123-muc-vpn1" }, { "source": "INTERNET", "target": "AZ2-testpermanent-123-muc-vpn2" }, { "source": "INTERNET", "target": "testPERMANENT-10.0.1.1" }, { "source": "INTERNET", "target": "testPERMANENT-10.0.1.2" }, { "source": "10.0.0.0/24", "target": "AZ1-testpermanent-123-muc-vpn1" }, { "source": "10.0.1.0/24", "target": "AZ2-testpermanent-123-muc-vpn2" } ] }

I think it works but not 100% because in the link I provided the generated graph is topdown but for example here LEVEL 1 is displayed at the bottom. And for example if I have groups, one node is displayed at the same level even though its one level lower. So I think it has to do something with groups and constraints.

JSON to image below: vpns are level 1 internet is level 2 etc. but if you look at LEAFS which are aligned with RS this should not be like that because they have different levels but grouping is a problem maybe

image

you can see without groups it works better but still not great image

marhyno commented 3 years ago

Added also step where nodes are sorted by levels when initiating simulation in diagram.ts

let preSortedNodes = data.nodes ? _.sortBy(data.nodes, "level") : [];
const nodes = preSortedNodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined));
marhyno commented 3 years ago

Further testing is showing me, its not reliable that the topology will show top down every time ... just small change will result in complete rebuild of topology, totally ignoring top down layout and inverting it or moving left to right, etc.

codeout commented 3 years ago

Thanks for the steps explained. Let me try to reproduce in my environment and investigate why inet-henge ignores levels defined in json.

marhyno commented 3 years ago

Thanks for the steps explained. Let me try to reproduce in my environment and investigate why inet-henge ignores levels defined in json.

Thanks, its not ignoring levels completely because If you drag one node, you are automatically moving all other nodes at that level, levels are just not aligned vertically. I tried different approaches and not sure what is causing it (maybe translate function ?) because I dont understand what can be difference between the approach I posted in link and my adjusted changed. E.g. I added new node above VPN nodes with LEVEL 5 all the way at the top and it messed up completely and showed first half of levels at the bottom. Keep me updated, I will try work on this too :)

marhyno commented 3 years ago

Any update ? I was looking at this, tried many different things and I found one solution. SetCola js Look at this - https://uwdata.github.io/setcola/ and choose kruger-foodWeb its perfectly aligned and top down and

I have created CodePen with WebCola + SetCola here https://codepen.io/marhyno/pen/xxRrLYZ (actual code start from LINE 782) and its working perfectly fine items are aligned top down and next to each other and you can create connections between the lowest level to the top without topology to change design. Can you look at it ? :)

codeout commented 3 years ago

I got the topology you mentioned finally.

image

source code: 598ba2f...4850f7f

A problem I found is that some of the constraints don't work, even they look valid to me. I'm looking into webcola now.

0: {type: "alignment", axis: "y", offsets: Array(2)}
1: {type: "alignment", axis: "y", offsets: Array(1)}
2: {type: "alignment", axis: "y", offsets: Array(2)}
3: {type: "alignment", axis: "y", offsets: Array(4)}
4: {type: "alignment", axis: "y", offsets: Array(2)}
5: {type: "alignment", axis: "y", offsets: Array(2)}
6: {type: "alignment", axis: "y", offsets: Array(2)}
7: {axis: "x", left: 0, right: 1, gap: 50}
8: {axis: "x", left: 3, right: 4, gap: 50}
9: {axis: "x", left: 5, right: 6, gap: 50}
10: {axis: "x", left: 6, right: 7, gap: 50}
11: {axis: "x", left: 7, right: 8, gap: 50}
12: {axis: "x", left: 9, right: 10, gap: 50}
13: {axis: "x", left: 11, right: 13, gap: 50}
14: {axis: "x", left: 12, right: 14, gap: 50}
15: {axis: "y", left: 0, right: 2, gap: 50, left_name: "testPERMANENT-10.0.1.1", …}
16: {axis: "y", left: 2, right: 3, gap: 50, left_name: "INTERNET", …}
17: {axis: "y", left: 3, right: 5, gap: 50, left_name: "AZ1-testpermanent-123-muc-vpn1", …}

// Something wrong
18: {axis: "y", left: 5, right: 9, gap: 50, left_name: "AZ1-testpermanent-muc-rs1", …}
19: {axis: "y", left: 9, right: 11, gap: 50, left_name: "10.0.0.0/24", …}
20: {axis: "y", left: 11, right: 12, gap: 50, left_name: "POD1-FCIPOCLEAF101", …}
marhyno commented 3 years ago

Hi, thank you very much, we found something like when it reaches certain level e.g. level 5 or 6 it stops using levels, and we dont know why. Bu thanks, maybe setcola would be helpful

codeout commented 3 years ago

So do you mean SetCola ( without inet-henge ) works for you? Or inet-henge can work along with SetCola? I'm just curious.

marhyno commented 3 years ago

Inet henge can work along set cola, if you look at the codepen I created, you can see that WebCola works with SetCola so I think it can work together with inet henge :)

The only thing which might cause problems is the setPosition function and translates, because if you look at the codepen, I did not use translate and nodes are perfectly aligned horizontally and vertically

codeout commented 3 years ago

Sounds great :tada: But yes, setPosition, translate or something like auto-layout stuff may conflict with SetCola. I'll read your codepen anyways. Thanks!

marhyno commented 3 years ago

Sounds great 🎉 But yes, setPosition, translate or something like auto-layout stuff may conflict with SetCola. I'll read your codepen anyways. Thanks!

Thanks, as always I will work on this too and provide results if any :)

marhyno commented 3 years ago

So far I have tried to remove .attr('transform', (d) => d.transform()); from group, node and link Added setCola to diagram.ts Added preprocessing by setCola in render function:

    var result = setcola
            .nodes(data.nodes)        // Set the graph nodes
            .links(data.links)       // Set the graph links
            .constraints(data.constraints)  // Set the constraints
            .gap(100)
            .layout();     

            const nodes = result.nodes ?
            result.nodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined)) : [];
            const links = result.links ?
            result.links.map((l, i) => new Link(l, i, this.options.meta, this.getLinkWidth)) : [];
            const groups = Group.divide(nodes, this.options.groupPattern, this.options.color);
            const tooltips = nodes.map((n) => new Tooltip(n, this.options.tooltip));

The json I use - added constraints 

{
  "nodes": [
  ],
  "links": [
  ],
  "constraints":[
    {
      "name": "leveled",
      "sets": {"partition": "level"},
      "forEach": [
        { "constraint": "order", "axis": "x", "by": "level", "gap": 150 },
        { "constraint": "align", "axis": "x" }
     ]
    },
    {
      "sets": ["leveled"],
      "forEach": [{ 
        "constraint": "order",
        "axis": "y", 
        "by": "level",
        "gap": 100,
        "order": ["carnivore", "herbivore", "plant"]
      }]
    }
  ]
}

But so far the generated topology looks like this

here

My latest update is here in this render part:

    this.cola.nodes(result.nodes)
                .links(links)
                .groups(groups)
                .constraints(result.constraints);
            this.setDistance(this.cola);

Here

marhyno commented 3 years ago

Anything new ? :)

marhyno commented 3 years ago

Hi, coming to you with working solution I think :)

After days of digging into cola, webcola, setcola, I finally managed to get it working. Magic lied in this line - this.cola.start(this.options.initialTicks,0, 0, 0);

The start() method now includes up to three integer arguments. First number will initially apply iterations of layout with no constraints, then iterations with only structural (user-specified) constraints and then iterations of layout with all constraints including anti-overlap constraints. Specifying such a schedule is useful to allow the graph to untangle before making it relatively "rigid" with constraints. So basically if I understand it right, the higher number the more space it occupies without "bumping" to each other

After adding setcola, including constraints into json, topology still looked somehow wrong. I compared it with the example I provided in codepen and found out ticks are the problem. If you leave it like above at 0,0,0 it generates random dynamic view. If you specify higher number, look below I put there 50,50,50 it generates nice looking perfectly aligned horizontal and vertical view. So I think you can include option into Diagram class if user wants to have hierarchical design, he needs to add TRUE so the ticks are different, and in json he must include constraints according to setCola page or look at the example at the bottom of this comment. Also as pointed in the earlier comment node.ts must also include level property to read it from json file. At the end of try catch part there is jQuery loop to remove additional nodes which are created by setcola and for end user are useless so I remove them.

var result = setcola
            .nodes(data.nodes)        // Set the graph nodes
            .links(data.links)       // Set the graph links
            .constraints(data.constraints)  // Set the constraints
            .gap(100)
            .layout();

            const nodes = result.nodes ?
            result.nodes.map((n, i) => new Node(n, i, this.options.meta, this.options.color, this.options.tooltip !== undefined)) : [];

            const links = result.links ?
            result.links.map((l, i) => new Link(l, i, this.options.meta, this.getLinkWidth)) : [];

            const groups = Group.divide(nodes, this.options.groupPattern, this.options.color);
            const tooltips = nodes.map((n) => new Tooltip(n, this.options.tooltip));   

            this.cola.nodes(nodes)
                .links(links)
                .groups(groups)
                .constraints(result.constraints);
            this.setDistance(this.cola);

            // Start to update Link.source and Link.target with Node object after
            // initial layout iterations without any constraints.
            this.cola.start(50, 50, 50);

            const groupLayer = this.svg.append('g').attr('id', 'groups');
            const linkLayer = this.svg.append('g').attr('id', 'links');
            const nodeLayer = this.svg.append('g').attr('id', 'nodes');
            const linkLabelLayer = this.svg.append('g').attr('id', 'link-labels');
            const tooltipLayer = this.svg.append('g').attr('id', 'tooltips');

            const [link, path, label] = Link.render(linkLayer, linkLabelLayer, links);

            const group = Group.render(groupLayer, groups).call(
                this.cola.drag()
                    .on('dragstart', this.dragstartCallback)
                    .on('drag', () => {
                        if (this.options.bundle) {
                            Link.shiftBundle(link, path, label);
                        }
                    })
            );

            const node = Node.render(nodeLayer, nodes).call(
                this.cola.drag()
                    .on('dragstart', this.dragstartCallback)
                    .on('drag', () => {
                        if (this.options.bundle) {
                            Link.shiftBundle(link, path, label);
                        }

                        Tooltip.followNode(tooltip);
                    })
            );

            // without path calculation
            this.configureTick(group, node, link);

            this.positionCache = PositionCache.load(data, this.options.groupPattern);
            if (this.options.positionCache && this.positionCache) {
                // NOTE: Evaluate only when positionCache: true or 'fixed', and
                //       when the stored position cache matches pair of given data and pop
                Group.setPosition(group, this.positionCache.group);
                Node.setPosition(node, this.positionCache.node);
                Link.setPosition(link, this.positionCache.link);
            } else {
                this.ticksForward();
                this.positionCache = new PositionCache(data, this.options.groupPattern);
                this.savePosition(group, node, link);
            }

            this.hideLoadMessage();

            // render path
            this.configureTick(group, node, link, path, label);

            if (this.options.bundle) {
                Link.shiftBundle(link, path, label);
            }

            path.attr('d', (d) => d.d()); // make sure path calculation is done
            this.freeze(node);

            const tooltip = Tooltip.render(tooltipLayer, tooltips);

            this.dispatch.rendered();

            // NOTE: This is an experimental option
            if (this.options.positionCache === 'fixed') {
                this.cola.on('end', () => {
                    this.savePosition(group, node, link);
                });
            }

            $('.node.rect').each(function(){
                console.log($(this));
                if ($(this).attr("id").indexOf("_set0_") > -1){
                    $(this).remove();
                }
            });

Add constrains from the code below in your json file which is loaded by inet-henge. These constraints are aligning items from top to bottom based on their level, levels are aligned ascended

{
  "nodes": [
      {
          "level": 4,
          "name": "AZ1-testpermanent-muc-rs1",
          "meta": {
              "name": "rs1",
              "hostname": "testpermanent-muc-rs1"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 5,
          "name": "AZ1-testpermanent-muc-rs2",
          "meta": {
              "name": "rs2",
              "hostname": "testpermanent-muc-rs2"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 4,
          "name": "AZ2-testpermanent-muc-rs3",
          "meta": {
              "name": "rs3",
              "hostname": "testpermanent-muc-rs3"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 5,
          "name": "AZ2-testpermanent-muc-rs4",
          "meta": {
              "name": "rs4",
              "hostname": "testpermanent-muc-rs4"
          },
          "icon": "/static/images/ios-xe.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF101",
          "group": "POD1",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF102",
          "group": "POD1",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF202",
          "group": "POD2",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 8,
          "name": "ACI-FCIPOCLEAF201",
          "group": "POD2",
          "meta": {
              "capabilities": ""
          },
          "icon": "/static/images/nexus9k.png"
      },
      {
          "level": 7,
          "name": "10.0.0.0/24"
      },
      {
          "level": 7,
          "name": "10.0.1.0/24"
      },
      {
          "level": 3,
          "name": "AZ1-testpermanent-123-muc-vpn1",
          "meta": {
              "name": "vpn1",
              "hostname": "testpermanent-123-muc-vpn1"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 3,
          "name": "AZ2-testpermanent-123-muc-vpn2",
          "meta": {
              "name": "vpn2",
              "hostname": "testpermanent-123-muc-vpn2"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 1,
          "name": "testPERMANENT-10.0.1.1",
          "meta": {
              "tenant-ip": "10.0.1.1"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 1,
          "name": "testPERMANENT-10.0.1.2",
          "meta": {
              "tenant-ip": "10.0.1.2"
          },
          "icon": "/static/images/vpn.png"
      },
      {
          "level": 2,
          "name": "INTERNET",
          "icon": "/static/images/internet.png"
      }
  ],
  "links": [
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-muc-rs1"
      },
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-muc-rs2"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF102"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF101"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF201"
      },
      {
          "source": "10.0.0.0/24",
          "target": "ACI-FCIPOCLEAF202"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-muc-rs3"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-muc-rs4"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF102"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF101"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF201"
      },
      {
          "source": "10.0.1.0/24",
          "target": "ACI-FCIPOCLEAF202"
      },
      {
          "source": "INTERNET",
          "target": "AZ1-testpermanent-123-muc-vpn1"
      },
      {
          "source": "INTERNET",
          "target": "AZ2-testpermanent-123-muc-vpn2"
      },
      {
          "source": "INTERNET",
          "target": "testPERMANENT-10.0.1.1"
      },
      {
          "source": "INTERNET",
          "target": "testPERMANENT-10.0.1.2"
      },
      {
          "source": "10.0.0.0/24",
          "target": "AZ1-testpermanent-123-muc-vpn1"
      },
      {
          "source": "10.0.1.0/24",
          "target": "AZ2-testpermanent-123-muc-vpn2"
      }
  ],"constraints":[
    {
      "name": "leveled",
      "sets": {"partition": "level"},
      "forEach": [
        { "constraint": "order", "axis": "x", "by": "name", "reverse": true, "gap": 150 },
        { "constraint": "align", "axis": "x", "reverse":true}
     ]
    },
    {
      "sets": ["leveled"],
      "forEach": [{ 
        "constraint": "order",
        "axis": "y", 
        "by": "level",
        "gap": 100,
        "reverse": true
      }]
    }
  ]

}

Really nice looking topology - here

codeout commented 3 years ago

Thanks! This is awesome :tada:

So I think you can include option into Diagram class

Yes, that's the last thing we have to do. I'd like to introduce a switch to plug setCola stuff into the original inet-henge, so that users can choose expected behavior - note that injecting constraints will cause extra CPU load in each calculation tick.

Besides that, we need to examine the magic number 50, 50, 50 is reasonable for other layouts. Or we should make them configurable maybe.

codeout commented 3 years ago

By the way, in your diagram mentioned here, do three groups overlap each other at the bottom, don't they?

marhyno commented 3 years ago

By the way, in your diagram mentioned here, do three groups overlap each other at the bottom, don't they?

Hi, yes they overlap only because I used both grouping and inner groups. This could also be solvable by changing behavior slightly be adding margin so we can see header of groups

Besides that, we need to examine the magic number 50, 50, 50 is reasonable for other layouts. Or we should make them configurable maybe.

Yeah I did not test it with higher or lower numbers