Adding a large number of connections takes a long time

tobiash-callone commented 1 year ago

I'm trying to visualize a bigger graph (1200 nodes and 2900 connections) with Draflow. In general it is working and the result is good. Unfortunately, I have the problem that adding connections between nodes gets slower and slower as the number increases. While the first few hundred connections are added quickly, it gets slower and slower until only a few connections are added per second, so that adding all nodes takes about 20 minutes. Is there a way to optimize this?

Looking forward to your feedback.

Best Regards Tobias

jerosoler commented 1 year ago

How do you add those nodes and those connections?

With the mouse UI? or by programming?

tobiash-callone commented 1 year ago

I add them by programming. But the browser is freezing and it takes up to 20min to execute the function addConnection(id_output, id_input, output_class, input_class) for all connections.

jerosoler commented 1 year ago

Hello

The problem is that it is updating the connections by the nodes. You could comment the following lines in function addConnection:

this.updateConnectionNodes('node-'+id_output);
this.updateConnectionNodes('node-'+id_input);

And delaying the execution of the connections.

In this example, it loads the nodes and loads the connections. Complete example:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Document</title>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/gh/jerosoler/Drawflow/dist/drawflow.min.css"
    />
    <script src="https://cdn.jsdelivr.net/gh/jerosoler/Drawflow/dist/drawflow.min.js"></script>
    <style>
      #drawflow {
        position: relative;
        width: 100%;
        height: 800px;
        border: 1px solid red;
      }
      </style>
  </head>
  <body>
    <div>
        <div id="drawflow"></div>  
    </div>
    <script>
    var id = document.getElementById("drawflow");

    const editor = new Drawflow(id);

    editor.addConnection = function(id_output, id_input, output_class, input_class) {
    var nodeOneModule = this.getModuleFromNodeId(id_output);
    var nodeTwoModule = this.getModuleFromNodeId(id_input);
    if(nodeOneModule === nodeTwoModule) {

      var dataNode = this.getNodeFromId(id_output);
      var exist = false;
      for(var checkOutput in dataNode.outputs[output_class].connections){
        var connectionSearch = dataNode.outputs[output_class].connections[checkOutput]
        if(connectionSearch.node == id_input && connectionSearch.output == input_class) {
            exist = true;
        }
      }
      // Check connection exist
      if(exist === false) {
        //Create Connection
        this.drawflow.drawflow[nodeOneModule].data[id_output].outputs[output_class].connections.push( {"node": id_input.toString(), "output": input_class});
        this.drawflow.drawflow[nodeOneModule].data[id_input].inputs[input_class].connections.push( {"node": id_output.toString(), "input": output_class});

        if(this.module === nodeOneModule) {
        //Draw connection
          var connection = document.createElementNS('http://www.w3.org/2000/svg',"svg");
          var path = document.createElementNS('http://www.w3.org/2000/svg',"path");
          path.classList.add("main-path");
          path.setAttributeNS(null, 'd', '');
          // path.innerHTML = 'a';
          connection.classList.add("connection");
          connection.classList.add("node_in_node-"+id_input);
          connection.classList.add("node_out_node-"+id_output);
          connection.classList.add(output_class);
          connection.classList.add(input_class);
          connection.appendChild(path);
          this.precanvas.appendChild(connection);
          //this.updateConnectionNodes('node-'+id_output);
          //this.updateConnectionNodes('node-'+id_input);
        }

        this.dispatch('connectionCreated', { output_id: id_output, input_id: id_input, output_class:  output_class, input_class: input_class});
      }
    }
  }

  editor.start();

  let lastId = 0;
  const nodes = [];
  for (let index = 0; index < 1500; index++) {

    const nodeId = editor.addNode('test1', 1, 1, Math.floor(Math.random() * 3000) + 1, Math.floor(Math.random() * 1600) + 1, 'test1', {}, '<div>Num:'+index+'</div>');
    if(lastId !== 0) {
    editor.addConnection(nodeId, lastId, "output_1", "input_1");      
    }
    nodes.push(nodeId);
    lastId = nodeId; 
  }

  function delay(time) {
   return new Promise(resolve => setTimeout(resolve, time));
  }

  nodes.forEach(async (node) => {
    await delay(0);
    editor.updateConnectionNodes('node-'+node);

  });  

</script> 
  </body>
</html>

converseKarl commented 1 year ago

I would also add to Jero's approach, splitting the flow as modules, or UI tabs, and building a connector flow in and out. we've done this and it works fine. This way you can modularize and keep flows much more sensible 70 to 120 nodes with connections is just fine per UI tab. If you've implemented the minimap it slows everything down.

One comment to jero is change the mini map to a block (square) you can drag around, as this is simpler but also speeds everything up significantly.

On the UI modules approach you can add, 20, 50, 100, 150+ tabs of 50, 70 , 150 nodes per UI tab/module with connections and in my view that's way more than enterprise standard. with good performance.

jerosoler / Drawflow

Adding a large number of connections takes a long time #672