Nested flows

[moklick]

The most wanted feature is nested flows. It would be great to have it in the next major release :)

[danieliser]

FWIW this seems to already be possible: https://codesandbox.io/embed/mjfiq

I found it when I was researching 3rd party examples in the wild.

[moklick]

haha :D who did this? It's nice but it is not really usable because the flows are separated and it's hard to setup and it doesn't work properly.

[mikebarkmin]

I have worked with Node-RED in the past. It is a flow-based application. Mainly for automation, but also REST-Services can be programmed. They have two approaches to "flows-in-flows".

First, you can define a custom node, aka. Subflow, which contains all the logic. You also declare what this node is called and how many inputs and outputs is has.

Second, you can link flows with are arranged in tabs by using the link node.

Documentation: https://nodered.org/docs/developing-flows/flow-structure

I think that the Subflow approach is better for React Flow. You can see on their user guide how Subflows work from a user interface perspective.

Based on that user guide, I would suggest the following behavior/user experience.

• A user can convert selected nodes into a subflow by pressing Ctrl+g (group shortcut in powerpoint). A popup will come up to ask the user for a name. All selected nodes in the parent flow will be replaced by the subflow node.
• A user can convert a subflow into regular nodes by pressing Ctrl+Shift+g (ungroup shortcut in powerpoint). The subflow node will be replaced by its nodes.
• A user can double-click a subflow will open it. Meaning, the current flow gets replaced by the subflow. There should be an indication, that one is in a subflow. There should also be a button to exit/close the subflow and return to the parent flow.
• A user can convert selected nodes inside a subflow into a new subflow. Meaning, subflows should be recursive.

[danieliser]

@moklick - The first version was better, he took it down and it broke the link on my list. So I reached out and he put back up what he remembered as he was just experimenting.

It is missing one key component now which is the disabling of dragging on the parent node when clicking on a child flow node. So yea its not functioning correctly, but I'm just happy to have the partially working example back up haha.

Fix was simple enough too, apply this technique here: https://codesandbox.io/s/react-flow-disable-drag-random-positions-wh082

---

As for working storage, well that is simply hooking up any changes in the child flow to sync to a single key on the parent nodes data object.

Otherwise it should work the way you have it now with those tweaks in place for that example.

I'm happy to clone that example and fix those 2 things so that its a working example, but I think the concept is totally doable now.

I'm all for improvements in the implementation if you see them.

---

I do like @mikebarkmin suggestion of a sub flow node, that would essentially just need to wrap the 2 fixes and any new custom logic into a node component and you'd be 90% there from what I've gathered.

[moklick]

Hey @danieliser, if that workaround works for you this is great :) but I would like to add simple API so that the devs don't need to nest ReactFlow containers.

---

@mikebarkmin thanks for you input. I think I won't add a UI for this but I would like to change the node API so that subflows are possible to visualize. What do you think about nodes and edges option for a node. With this we could do unlimited nesting of sub flows. I am not sure about the connections. Which node inside a subflow gets the input of the outer flow? How to define this? With reaflow it's possible to define nested nodes. They define a parent for a node or edge.

[mikebarkmin]

Even if you're not planing a UI for this, I would prefer the grouping behavior I have described. This you mean, that a new node type would be needed.

type: "subflow" (or "group")
data: {
   name: string;
   nodes: FlowNodes[],
   edges: FlowEdges[],
   inputs: string[] (node ids),
   outputs: string[] (node ids),
}

For each input, a Handler would be created on the left-hand-side of the node. For each output, a Handler would be created on the right-hand-side of the node. If this is to complex it could be simplified to one input and one output.

[danieliser]

Not sure why you would need to reference nodes, edges, entry and exit nodes as separated properties.

React flow accepts an array of nodes and edges already and is able to properly identify the entry and exit nodes when rendering.

I can see one possibility for wanting to know entry and exit ids and counts from within the sub flow and that is if you wanted to let the parent node offer a matching number of input/output handles each linked to a specific sub flow entry/exit node.

Even then storing the inputs/outputs and list as separate data properties is only a micro optimization to prevent need to quickly filter the full flow array each time and can probably be cached and handled internally as opposed to exposed as a saveable data point.

A sub flow node should basically accept nearly identical properties as the main ReactFlow component so that someone can reasonably expect each flow to work the same reguardless of location.

I do like the idea of a subflow node though that contains all the logic for calculating and mapping entry/exit to input/output etc.

[datoslabs]

@moklick and @danieliser, I think having a subflow node type acting as parent (container) property for a group of children nodes (similar to reaflow) is a more versatile approach. As an example:

[
  { id: '1', type: 'input', data:{...}},
  { id: '2', type: 'default', data:{...}},
  { id: '3', type: 'subflow', data:{...}},
  { id: '3.1', type: 'default', parent:'3', data:{...}},
  { id: '3.2', type: 'default', parent:'3', data:{...}},
  { id: '4', type: 'default', data:{...}},
  { id: '5', type: 'output', data:{...}},
  { id: 'e1-2', source: '1', target: '2'}
  { id: 'e2-3', source: '2', target: '3'}
  { id: 'e3.1-3.2', source: '3.1', target: '3.2'}
  { id: 'e3-4', source: '3', target: '4'}
  { id: 'e4-5', source: '4', target: '5'}
]

Which node inside a subflow gets the input of the outer flow?

This can either be implicit or explicit which users/developers can decide based on their requirements. Above example is an implicit approach, ie node 3.1 gets the input of the outer flow and node 3.2's output is piped into node 3's output. An explicit approach of the above example will essentially replace edges e2-3 with { id: 'e2-3.1', source: '2', target: '3.1'} and e3-4 with { id: 'e3.2-4', source: '3.2', target: '4'}

A sub flow node should basically accept nearly identical properties as the main ReactFlow component so that someone can reasonably expect each flow to work the same reguardless of location.

While having a sub flow node that accepts nearly identical properties as the main ReactFlow component makes it easier to optimize for performance, my fear is it would be more difficult to implement complex interactions between sub flows and parent/other sub flows. For example, if there is a requirement to model the sub flow as async processes with multiple inputs and outputs from the parent flow (or another sub flow) like the following example from Mermaid.js:

A sub flow node implementation that basically acts as a separate ReactFlow component will make it difficult to expose the internal nodes for connection.

[danieliser]

Maybe I'm missing something, but I'm not quite sure your example really fits the true intention of a subflow, and if I were setting it up to accomplish the image you included I'd likely suggest that one and three aren't subflows, but rather multiple complex entries to the same "main" flow. There is no reusability in having those 2 declared subflows. They seem more like categorization of processes, such as these tasks are IT, and these are Marketing etc. But again not really subflows. and doesn't really warrant a special subflow, as opposed to some customized tagging of nodes & rendering with colors.

two on the other hand is a self contained flow on its own, with its own entry and exit points.

FWIW, the way the nodes are stored internally isn't of major concern to me, I see some upside in your flattening of the node list in being able to access all points from anywhere easily, but it again taints the purpose of subflows in my mind. And there are easy ways to do that without redefining the concept.

For example one app I've used this library on we maintained the flow state outside of the components entirely, and we stored it in a nested object for simpler processing on the other end since it was always a one way decision flow. As such we just made 2 conversion functions that convert nested to flat and back again. We convert it on the way in and out.

If there were simply a couple helper functions included with the react-flow-renderer library to get a flat or nested representation, it wouldn't really matter how it was stored in the internal state.

Something else to consider, if he did include the dynamic entry handle counts based on subflow entry points, there would already be logic for subflows to have setups with multiple varying entries, but that may be difficult to do with flat node lists, mainly because you would then have subflow entry/input nodes that would need to be differentiated from true top level parent entry/inputs. Point being it would likely mean nearly all of the existing node processing loops would have to be refactored to handle is this a parent/sub/grandchild etc and react accordingly.

Doing it nested means all that existing logic stays the same. A single new node component can manage most of the heavy lifting of making a flow and subflow interact by sitting right in the middle of them. And I'm quite confident that most all of the features discussed could be done using the existing node customizations available like dynamic handle creation etc.

Just my 2 cents.

[datoslabs]

I believe we need both flatten and nested paradigms as they facilitate different types of use cases. But allow me to address the example first. Admittedly, the example I took from mermaid.js is a bit over simplified; however, if I make a few additions as follows, it would help to see why it makes sense to make one and three as subflows.

Subflows one and three run concurrently but a2 will only start when a1 and c1 are both completed plus a1 and c1 will only instantiate and run when subflows one and three are triggered/started respectively. When validation fails, only subflow three is re-executed. Flattened representations like this have the following advantages:

1. Simplified workflow diagram.
If a1, a2, c1, c2 are all part of the main flow, users will need to introduce additional flow logic to achieve the same effect. In my opinion, subflows represent independent execution contexts with separate variables and flow logic; hence, it offers execution isolation in addition to improved reusability. Therefore, with one and three being separate subflows, the workflow diagram is simplified whether the user wants to only re-execute three or both one and three.

2. Simplified subflow interactions.
In this example, it would help to treat c1-a2 edge as an event/observer relationship (ie via message producer-consumer) especially because a2 and c1 belong to separate subflows. In a Nested representation, inter workflow connections have to go through the parent workflow/subflow (ie. c1 - one then one - a2); Flattened representation on the other hand allow users to connect c1 and a2 directly. Therefore, for "intra"-subflows (ie subflows that are reusable only within a main flow), the Flattened representation is simpler and more suitable to use; whereas "inter"-subflows (ie subflows that are truly reusable and shared with multiple workflows), Nested representation will be more suitable. We should note that in Nested representation, users are required to memorialize all input and output handles via the parent workflow definition to facilitate connection from external workflows.

3. Ease of revision.
Let's suppose the user initially did not see the need to separate a1, a2, c1, c2 into subflows and kept them as part of the main flow in "version 1"; however, due to the new requirements, the user revises the design and separated c1, c2 into a subflow in "version 2". Since workflows can be tweaked often, we would want to make it effortless for users to create or flatten subflow like what @mikebarkmin suggested with Ctrl-g and Ctrl-shift-g. This can be easily achieved in Flattened representation while in Nested representation would most likely re-render the screens as well as requiring id collision detection and handling requirements (e.g. when both the main flow and the nested sublow have element id=1, since users can add elements and edges independently, the "merge" process needs to detect and reassign duplicate element and edge IDs ).

[danieliser]

@datoslabs - Appreciate the explanation but in my mind, subflows don't have entries in the middle, such as the links from c1 -> a2 and two to c2.

Agree supporting both flat & nested would be good, but it really has no bearing on internals, just wishful thinking in terms of exposed API methods to get the data preformatted a specific way, something we currently do after retrieving the data from the flow.

[mikebarkmin]

I think that the drawing of @datoslabs is a bit misleading. Here is how I think about it, which should work, if we explicitly name the input and output nodes of a subflow.

Subflow "one" has three handles: Two target (a, b) ones and a source (c) one. Subflow "three" has four handles: Two target (a, b) and two source (c, d) ones. Subflow "two" has four handles: Two target (a, b) and two source (c, d) ones.

These are the edges:

• edge-1: start -> one-a
• edge-2: three-c -> one-b
• edge-3: start -> three-a
• edge-4: two-c -> three-b
• edge-5: three-d -> two-a
• edge-6: two-d -> valid
• edge-7: one-c -> two-b
• ....

I also think that nested subflows should be support, and therefore I suggest the following structure, which is an update of my previous one, but should respect most comments:

type SubflowNode = {
  type: "subflow";
  data: {
    label: string;
    sources: Record<handleId, nodeId>;
    targets: Record<handleId, nodeId>;
    flow: ReactFlow
  }
}

[datoslabs]

@danieliser:

Appreciate the explanation but in my mind, subflows don't have entries in the middle, such as the links from c1 -> a2 and two to c2.

I have encountered many scenarios where a reusable sub workflow starts and needs to wait for additional inputs from other external async/concurrent processes (like c1->a2) mid-stream. My intention is to broaden the types of use cases that can be supported by the new subflow feature to help this project gain even more adaption from a thriving community.

To summarize, I think there are 2 types of subflows that have been proposed - embedded (aka flattened from the earlier part of this thread) vs external (aka nested):

1. Embedded subflows are reusable subflows that do not have declarative inputs and outputs. In some ways, it resembles inner classes in an OO paradigm where its internals are open to the encapsulating/parent flow. Embedded subflows only exist as part of the encapsulating/parent flow.

2. External subflows are reusable flows that can exist independently outside of the encapsulating/parent flow and as such require declarative inputs and outputs. External subflows can be rendered in whole by a <ReactFlow> graph. Unless I misunderstood, I believe the use cases @danieliser and @mikebarkmin fall into this category.

[xpluscal]

Hi! Is there any progress on this? Anyway I can help?

[moklick]

Hey everyone! Thanks for all the input :) This really helps. We are back in October and have more time for React Flow hopefully ;) The plan is to release a new major version until the end of the year.

[totaland]

hi @moklick can we able to do the nested node now?

[leejustin]

hi @moklick can we able to do the nested node now?

No. This issue is still open. moklick is really good about closing out issues as well as clear changelogs: https://github.com/wbkd/react-flow/releases

[truong-hua]

@moklick how do you think about the group isolation feature like Adobe Illustrator. So users can isolate a group a do editing only items inside that group. That would make the editing job scalable no matter how many nodes.

[moklick]

We are currently working on this feature. Which API do you prefer? We have a little voting here: https://github.com/wbkd/react-flow/discussions/1024#discussioncomment-1512393

[pgupta56]

when is v10 releasing ??