Build computational graph with interactive library

[sc420]

Functional Spec

Interactive UI

• Can add nodes to the Graph canvas through the "Add nodes" panel
• Can connect the nodes on the canvas
• Can update the values for variable nodes

Use Cases

• Can switch differentiation mode/target
• Can show the updated f values when changing node values
• Can show the updated derivatives when changing node values

[sc420]

Looks promising: https://reactflow.dev/

Consider to replace react-diagrams with it because it has richer docs and better look and feel. Please try replacing the current react-diagrams with it to see if it's acceptable.

[sc420]

I decided to use React Flow because the docs and examples are clear. For future references:

• Drag and drop new node: https://reactflow.dev/docs/examples/interaction/drag-and-drop/
• Custom nodes: https://reactflow.dev/docs/guides/custom-nodes/
• Multiple handles aligned with textboxes: https://reactflow.dev/docs/examples/overview/
• Connection validator for cycle prevention: https://reactflow.dev/docs/examples/interaction/validation/
• Interaction properties: https://reactflow.dev/docs/examples/interaction/interaction-props/
• Save and restore: https://reactflow.dev/docs/examples/interaction/save-and-restore/
• Minimap: https://reactflow.dev/docs/api/plugin-components/minimap/

For look and feel, we can check out the following websites as the references:

• Datablocks (big handles): https://editor.datablocks.pro/demo
• LangFlow (accordions in side panel): https://www.youtube.com/watch?v=GoNJiQxKDK0

Demo:

[sc420]

Design Spec

React Architecture

classDiagram
    MainContainer --> FeaturePanel
    FeaturePanel --> AddNodePanel
    FeaturePanel --> EditNodesPanel
    FeaturePanel --> LoadSavePanel
    AddNodePanel --> Operation
    MainContainer --> GraphContainer
    GraphContainer --> Graph

    Graph ..> Operation
    Graph ..> VariableNode
    Graph ..> ConstantNode
    Graph ..> OperationNode

    class MainContainer {
        - graphState: GraphState
    }

    class FeaturePanel {
        - graphState: GraphState
    }

    class AddNodePanel {
        - operations: features.Operation[]
        - builtInOperations: features.Operation[]
    }

    class EditNodesPanel {
        - graphState: GraphState
    }

    class LoadSavePanel {
        - graphState: GraphState
    }

    class Operation {
        <<features.Operation>>
        - id: string
        - fCode: string
        - dfdyCode: string
        - inputPorts: string[]
        - helpText: string
    }

    class GraphContainer {
        - graphState: GraphState
    }

    class Graph {
        <<react-flow.Graph>>
        + ReactFlowGraph(operations: features.Operation[])
        - graph: graph.Graph
    }

    class VariableNode {
    }

    class ConstantNode {
    }

    class OperationNode {
    }

The GraphState contains everything about the current graph state. It records a list of built-in/custom operations, graph nodes/edges/connections, react-flow node positions, etc.

When the list of operations (features.Operation) is updated, the Graph (react-flow.Graph) may not use it immediately but should store it for later use. When a node is dragged and dropped to the canvas, it can find the necessary data in the previous recorded list of operations.

There're only 3 types of nodes we can add: variable, constant and operation. The drag-and-drop data should indicate the type and the operation id (if the type is operation). The UI of VariableNode, ConstantNode and OperationNode should be designed separately. OpeartionNode will be heavily reused because it may contain any type of operations. We may even support tensor types in the future (and value should be matrix). OperationNode will need to create input ports dynamically, please note that doc say you need to use some hook to notify the changes.

We don't rely on React context or third-party state management libraries because props are easier to test.

New directories:

• features: Features that are existed because of the use cases, don't rely any React Flow stuff in it
• react-flow: Components that use React Flow library, may be replaced in the future

[sc420]

Design Spec v2

React Architecture

classDiagram
    App --> Title
    App --> Sidebar
    App --> GraphContainer
    GraphContainer --> FeaturePanel
    GraphContainer --> Graph
    GraphContainer --> GraphToolbar
    GraphContainer --> GraphStateController
    FeaturePanel --> AddNodePanel
    FeaturePanel --> EditNodesPanel
    FeaturePanel --> LoadSavePanel
    AddNodePanel --> Operation

    Graph ..> Operation
    Graph ..> VariableNode
    Graph ..> ConstantNode
    Graph ..> OperationNode

    class GraphContainer {
        - selectedFeature: SelectedFeature
        - graphStateController: GraphStateController
        - nodes: reactflow.Node[]
        - edges: reactflow.Edge[]
    }

    class GraphStateController {
        + getReadOnlyState(): ReadOnlyGraphState
        + getReadWriteState(): GraphState

        + addNode(nodeType: string, nodes: Node[]): Node[]
        + selectNode(nodeId: string, nodes: Node[]): Node[]
        + removeNode(nodeId: string, nodes: Node[]): Node[]
        + resetNodes(): [Node[], Edge[]]
        + load(data): Node[]
        + save(nodes: Node[], edges: Edge[]): void
    }

    class FeaturePanel {
        - selectedFeature: SelectedFeature
        - graphState: ReadOnlyGraphState
    }

    class AddNodePanel {
        - operations: features.Operation[]
        - builtInOperations: features.Operation[]
    }

    class EditNodesPanel {
        - graphState: ReadOnlyGraphState
    }

    class LoadSavePanel {
        - graphState: ReadOnlyGraphState
    }

    class Operation {
        <<features.Operation>>
        - id: string
        - fCode: string
        - dfdyCode: string
        - inputPorts: string[]
        - helpText: string
    }

    class Graph {
        <<react-flow.Graph>>
        - graph: graph.Graph
        - graphState: GraphState
    }

    class VariableNode {
    }

    class ConstantNode {
    }

    class OperationNode {
    }

When the user clicks the list item in AddNodesPanel, we should add a node on the Graph. In previous design spec, we don't know who should be responsible to update the state (AddNodesPanel? MainContainer? GraphContainer? Graph?).

It seems that FeaturePanel is closely related to Graph, so why not put them closer?

In this new design spec, we make the architecture more flat by removing MainContainer and putting FeaturePanel and Graph closer. GraphControl should be renamed to GraphToolbar to avoid confusion with GraphStateController.

The data flow is like this when user invokes something in the feature panel:

1. Child feature panel propagates the event to the parent GraphContainer
2. GraphContainer calls corresponding API in GraphStateController to update the graph state
3. Graph will re-render when the graph state updates automatically

Note that graph state resides in GraphContainer so that React knows we update them. GraphStateController only contains logic to update the graph state.

[sc420]

Datablocks multiple input ports example:

threegn example:

chaiNNer example:

[sc420]

We should rename core graph classes/files to have shorter names, non-core react/react flow stuff should have longer names. It can avoid name collision and confusion. E.g., Graph -> ReactFlowGraph

[sc420]

Design Spec v2.1

Architecture

classDiagram
    App --> Title
    App --> Sidebar
    App --> GraphContainer
    GraphContainer --> FeaturePanel
    GraphContainer --> ReactFlowGraph
    GraphContainer --> GraphToolbar
    GraphContainer --> CoreGraphController
    GraphContainer --> ReactFlowController
    FeaturePanel --> AddNodePanel
    FeaturePanel --> EditNodesPanel
    FeaturePanel --> LoadSavePanel
    AddNodePanel --> FeatureOperation
    ReactFlowGraph --> CustomNode

    class GraphContainer {
        - coreGraphController: CoreGraphController
        - reactFlowController: ReactFlowController
        - coreGraph: Graph
        - nodes: reactflow.Node[]
        - edges: reactflow.Edge[]
        - featureOperations: FeatureOperation[]
    }

    class CoreGraphController {
        + addNode(nodeType: string, graph: Graph): Graph
        + removeNode(nodeId: string, graph: Graph): Graph
        + connect(node1Id: string, node2Id: string, node2PortId: string, graph: Graph): Graph
        + disconnect(node1Id: string, node2Id: string, node2PortId: string, graph: Graph): Graph
    }

    class ReactFlowController {
        + addNode(nodeType: string, featureOperations: FeatureOperation[], nodes: Node[]): Node[]
        + dropNode(nodeType: string, position: XYPosition, featureOperations: FeatureOperation[], nodes: Node[]): Node[]
        + changeNodes(changes: NodeChange[], nodes: Node[]): Node[]
        + changeEdges(changes: EdgeChange[], edges: Edge[]): Edge[]
    }

    class FeaturePanel {
        - selectedFeature: SelectedFeature
        - featureOperations: FeaturesOperation[]
    }

    class AddNodePanel {
        - featureOperations: FeaturesOperation[]
    }

    class EditNodesPanel {
        - graphState
    }

    class LoadSavePanel {
        - graphState
    }

    class FeatureOperation {
        - id: string
        - text: string
        - operation: Operation
        - inputPorts: Port[]
        - helpText: string
    }

We should only initialize core graph once in useEffect, otherwise the React could call its method twice and cause side effect because we don't dee copy our core graph class.

We split the responsibility into two classes, the core controller and react flow controller. We should update core controller first, and update react flow controller based on the output of core controller.

[sc420]

postpone the milestone by one week because QA is important