Creating many cluster in the same time

[Manerial]

Hi all,

I've got an issue: When I try to create many clusters in the same function. It works when I try to create them only with "normal" nodes, but when one of them has to include a cluster that must be created in the same function, it doesn't work.

Example:

var rank = 0;
// cluster_infos contains a cluster_Id and a rank. The rank of the cluster is the maximum rank of the nodes it contains +1. A "normal" node is -1. {"cluster0" : 0, "cluster1" : 1}
while(redoClustersByRank(cluster_infos, rank)) {
    rank++;
}

function redoClustersByRank(cluster_infos, rank){
    var rankDone = false; // is returned at the end
    for (var cluster_i in cluster_infos) {
        if(cluster_infos[cluster_i] == rank) {
            rankDone = true;
            var nodeCluster = graphData["nodes"]["_data"][cluster_i]; // just a structure that I use to store clusters data to use in clusterNodeProperties of clusterOptionsByData
            var clusterOptionsByData = {
                "processProperties":function (CLUSTER_OPTIONS, childNodes, childEdges){
                    var totalMass = 0;
                    for (var node in childNodes){
                        totalMass += childNodes[node].mass;
                    }
                    CLUSTER_OPTIONS.mass = totalMass;
                    CLUSTER_OPTIONS.scaling = {
                        min:15 * totalMass,
                        max:15 * totalMass
                    }
                    return CLUSTER_OPTIONS;
                },
                "joinCondition" : function (childOptions){
                    return childOptions["cluster_id"] == nodeCluster["id"];
                },
                "clusterNodeProperties" : {
                    "id":nodeCluster["id"],
                    "x":nodeCluster["x"],
                    "y":nodeCluster["y"],
                    "hidden":nodeCluster["hidden"],
                    "color":nodeCluster["color"]["background"],
                    "label":nodeCluster["label"],
                    "borderWidth":3,
                    "shape":'database'
                }
            };
            graphCanvas.cluster(clusterOptionsByData);
        }
    }
    return rankDone;
}

When I do that manually (one by one), it works. But using that code, it just creates the rank 0 nodes but not the rank 1 and more.

Did I understand something wrong or I just can't do that by this way?

[wimrijnders]

Congratulations, you have just invented 'Cluster Inception'. I'm not sure if I have to classify this as Problem or Feature Request.

I know where this comes from: for reasons of efficiency, the joinCondition results are batched and the state is updated in one go after the operation. Because of this, any created clusters you are re-using in clusters may not be available for clustering afterward.

The only workaround I can think of now, is to do the clustering in steps: first run the cluster operation to add the clusters you want to cluster; then do a second clustering operation to cluster those.

As for a 'fix', I really have to think hard about this, this is very meta. Please bask in the knowledge that you are the very first person to think of doing this.

[AndrewMut]

I didn't get full measure of the tragedy but, cluster really can't be re-used so you can store cluster info in some node, I am storing my cluster's info in parent node from which I call 'cluster by', you may create invisible node for each cluster you create and store it there. Going back to the state of network init, here you may want to collapse something as it was before refresh so you may keep cluster info in some invisible node.

It's not the same but here is an example of how I am closing clusters, and there is no matter if cluster inside cluster or cluster inside cluster inside cluster and so on.

function initClusters(clusterId) {
      var allNodesInData = data.nodes.get();
      var nodesWithInClusterTrue = [];

      function collapseCluster(value) {
        // value here is full node object
        var optionToCombine = 'clusterId_' + value.meta.CID;
        var clusterOptionsByData = {
          joinCondition: function(childOptions) {
            return childOptions.meta[optionToCombine] === true;
          },
          clusterNodeProperties: {
            id: 'clusterId_' + value.meta.CID,
            label: value.text + ' Cluster',
            borderWidth: 1,
            image: {
              url: value.image.selected, //TODO check it
              selected: value.image.selected,
              unselected: value.image.unselected
            },
            brokenImage: '/assets/images/broken-image.jpg',
            size: value.size,
            shape: value.shape,
            color: {
              hover: {
                border: '#00e676',
                background: value.color
              },
              background: value.color,
              highlight: {
                background: value.color.highlight.background
              }
            },
            shapeProperties: value.shapeProperties,
            labelHighlightBold: false,
            borderWidthSelected: 1,
            x: value.position.top,
            y: value.position.left,
            meta: value.meta,
            parentNodeId: value.id
          }
        };

        network.cluster(clusterOptionsByData);
        var clusteredNodeId = 'clusterId_' + value.meta.CID;
        network.clustering.updateClusteredNode(clusteredNodeId, {
          color: {
            hover: {
              border: '#00e676',
              background: value.color
            },
            background: value.color,
            highlight: {
              background: value.color.highlight.background
            }
          }
        });
      };

      angular.forEach(allNodesInData, function(value, key) {
        if (value.meta.inCluster === true) {
          nodesWithInClusterTrue.push(value);
        }
      });

      nodesWithInClusterTrue.reverse();

      angular.forEach(nodesWithInClusterTrue, function(value, key) {
        collapseCluster(value);
      });
    };

[Manerial]

I understand. For the moment, I'll look for an other solution and if I find something good, I'll tell you. Thanks a lot for your answer, and I hope we will find a solution!

[AndrewMut]

@Herment , could you please try to explain for me what and for what purposes you are doing? I didn't get what are you trying to do. May be will find some solutions, @wimrijnders is really good in that.

[Manerial]

Ok, so:

I use Vis.js to display a network graph. Thanks to different functions, I can create clusters easily by selecting them. Thanks to that, I can create: clusters of nodes, clusters of nodes and clusters, clusters of clusters.

All these clusters are saved in a database line the nodes, so they are considered like them. But when I want to reload my graph, I need to re-create all of these clusters in a single function to directly find back all the clusters I've done.

To do that, I've created my own function, "redoClusters", that takes all clusters in my database and look the rank of each of them. Thanks to that, I can create all cluster of nodes, but not Clusters that contains clusters.

More than that, I can open all clusters I want so I use "redoCluster" to redo them without reloading the graph (doing this will loose all changes I've done).

[AndrewMut]

Got it. I've got the problem with 'redoClusters' on reload. And @wimrijnders was right saying that you should do this in few iterations. As I see that, you shoud store info about your current clusters somwhere. As mentioned before, may be try to collect this info in invisible nodes by adding to the nodes hidden : true.

So the plan is:

1. On cluster collapse create invisible node with unic ID and property 'inCluster : true' for future (It will be used just as data storage for network purposes) a. All nodes that you want to collapse into this cluster also should have that id as property like 'unic ID = true', in your case it looks like you should use 'collapse by rank1 = true'
2. Create simple function 'CollapseBy'. You will be collapsing by unic cluster ID or rank.

   
   var clusterOptionsByData = {
         joinCondition: function(childOptions) {
           return childOptions.meta[unicID] === true;
         }

network.cluster(clusterOptionsByData);

3. In light that you're using ranking you should go through all network nodes with lower to higher rank and collapse them one by one. 

Let me know if it will help somehow. Otherwise I can share with you some of my code for similar purposes. Unfortunately project I am working with not open source and I can't share the full code.

[Manerial]

Thanks, but I'm afraid I didn't understand all you said.

What I don't understand is:

• Why do you use invisible nodes? Is it to have a physical representation of the cluster inside the upper LVL cluster (that's what I understand)?
• I don't understand:

  In light that you're using ranking you should go through all network nodes with lower to higher rank and collapse them one by one. Because if I do that, I have to do it manually, and that's not what I want.

Can you please give me more details about these points? Thanks

[AndrewMut]

With pleasure.

Why do you use invisible nodes? Is it to have a physical representation of the cluster inside the upper LVL cluster (that's what I understand)?

I am not using then, I store them inside my parent node.

as you can see I use buttons dirrectly on the node wich will become cluster. So all cluster info inside it. As I can understand you are using side buttons to collapse something, so you should have some storage for cluster info in network instance. For future usage on reload. (this invisible node also should be stored somewhere outside), and you got it right - physical representation of the cluster.

Can you please give me more details about these points? If i understand you right, your regular nodes have rank = 1, clusters rank = 2, cluster in cluster rank = 3 etc. So first itteration should collapse everething with rank 1, and result of that should be cluster with rank 2, second itteration should check network if there anything with rank 2 and collapse all items with that rank, etc.

Because if I do that, I have to do it manually, and that's not what I want. You shouldn't, just like that:

$q.when(collapseRank1)
.then(collapseRank2)
.then(collapseRank3)

and so on.

[Manerial]

I'll try that. Thanks

[wimrijnders]

@AndrewMut Thanks a million for your effort here. You are on my beer list from now on, meaning that if I ever meet you, the drinks are on me!

[wimrijnders]

So, I see two issues here:

1. Recently created clusters can't be clustered dynamically. This I can do something about.
2. Cluster info can not be stored externally. This is what @AndrewMut addressed above.

I can do something about 1. I hope 2 has been handled sufficiently for you by @AndrewMut (thanks!). I still don't know if I should classify this as a bug or feature request.....but I will do a fix regardless. Keeping the issue open for this.

[Manerial]

I was thinking and remembering that Problem 2 was not a problem for me because I stored all my nodes information (even clusters) in a specific variable. On the other hand, I can not solve problem 1. I will wait for the new version, when it will be rectified. Thanks a lot to both of you!

[wimrijnders]

My pleasure!

I will wait for the new version, when it will be rectified.

I'm afraid that it won't make it into the upcoming release, which is being built as we 'speak'. Hopefully it will be in the release after that, if I or anyone else gets around to it - so much to do.....

[Manerial]

I understand, no worries :-) Could you give me an idea of when the new version (with this fix) will be delivered?

[wimrijnders]

Since the current release is going out now, that would be in about three months time plus or minus, uhm, a lot.....really, depends on the availability of the maintainers. Sorry to be so imprecise.

[Manerial]

Ok, thank you. I'll avoid cluster of clusters creation during this time, and allow it in more or less three months!