Feature/lineage graph

[HX-Tfd]

Primary changes:

1.Implemented action/data graph separation(#74).

• We can access the data, action and the full graph separately though the drop-down button in the LineageTab.
• To achieve action and data object separation, we first simplify the input data when we create the graph: all M:1 actions (with M-1 extra nodes and extra edges) are merged to a single actionNode with multiple incomming edges. All duplicate edges and nodes of M:N actions are removed after merging. We do not perform merging when creating the data or action graph, as we create them from the already merged full graph. The current algorithm is implemented in a straightforward fashion. Future optimization of the separation/merge algorithm possible.
• The full graph is prepared once in src/util/ConfigExplorer/ConfigData.ts and the data and action graphs are created the first time we click on the corresponding view in the drop-down button
• There is a subtlety in the graph view switching: when we switch to the action graph view in a data graph view, the UI displays the first data node in the levelOneNodes of the data graph, and the same logic applies when we switch to the data graph view in a action graph view. This could, maybe, be replaced by a direct neighbour, depending on the need.

2.Implemented #76.

• The custom node and edge components can be enabled by doing the following:

  • making custom node and edge components available in the lineage tab by adding (in LineageTabWithSeparateView):

    const nodeTypes = {
    customDataNode: CustomDataNode,
    }
    const edgeTypes = {
    customEdge: CustomEdge,
    };

  • setting type: 'customDataNode' in the createReactFlowNodes function,
  • setting type: 'customEdge', in the createReactFlowEdges function,
  • and nodeTypes={nodeTypes} edgeTypes={edgeTypes} in the Reactflow tag in LineageTabSep. Note that we don't use custom edges, but smoothStep instead. Otherwise the edges get messy.
• Some of the components in the custom node are disabled (e.g. progress bar, last run status etc.). They will be gradually added depending on further discussions.
• If we need more information to be displayed on the node, just add more fields to the flowProps interface in LineageTabWithSeparateView and access it in the custom node component. The additional props can be passed in src/components/ConfigExplorer/ElementDetails.tsx where the LineageTabSep is opened.

3.Refactored components of LineageTab into a folder.

• The folder consists of LineageGraphComponents (Custom Nodes and Edges),LineageGraphToolbar(reactflow components refactored from the old LineageTab) and the old (LineageTab) and newer version (LineageTabWithSeparateView) of the LineageTab.

Minor changes:

1.Disabled the open lineage button in Connections in the ConfigurationTab, as connections don't have a lineage (and opening the tab would cause an error). 2.Added several Unit Tests for the Graph class (in the tests folder). The imports are updated correspondingly. 3.Added/removed comments and TODOs

Potential problems

1.In the current layout, the edges still overlap with the nodes sometimes 2.What information should we be displaying on the node?

Incomming Features

1.Visualizing the statuses of the nodes of a single run. For the moment, only the statuses of the last runs are shown. However, this could make one falsely believe that the displayed status icons are all from the same last run. We can consider implementing new drop-down bar to visualize different relevant information of the (same) last runs. 2.Making the LineageTabToolbar draggable

Future Considerations

1.We should consider not to re-render the entire graph if the view has not changed. This should also be addressed when we implement graph expansion/compression, where neighbour nodes can be added/removed gradually. 2.We should maybe re-activate the open lineage button for the connections, in which we show the lineage graph that belongs to the entire connection.

[HX-Tfd]

Small improvements according to demo:

• recenter graph to selected object on: change of object, change of layout, expand/collapse
• increase edge corner radius
• fix icons of action/dataobject graph, fix display selected graph type icon
• reduce vertical gap between objects to 2/3 of current value in vertical graph
• reduce horizontal gap between objects to 2/3 of current value in horizontal graph
• make edges selectable (only one at a time) and highlight selected edge by a different color
• increase size of edge arrow

Bigger improvements to analyze (dont implement yet...)

• improve minimap contrast/coloring
• make objects horizontally resizable (all objects should resize at once), or suggest a different solution for not abbreviating names...

[zzeekk]

Two comments about "future considerations":

1.We should consider not to re-render the entire graph if the view has not changed. This should also be addressed when we implement graph expansion/compression, where neighbour nodes can be added/removed gradually.

Re-render is ok, but we should no re-layout.

2.We should maybe re-activate the open lineage button for the connections, in which we show the lineage graph that belongs to the entire connection.

No, connection lineage makes no sense. You can already select all DataObjects with the same connection, thats enough...