Top-down design approach

[trustmaster]

As we have figured out in a series of discussions on the FBP mailing list, one of the major problems with NoFlo-UI for designers of data processing systems, including large heterogeneous systems, is that it currently only supports Bottom-up approach, while engineers use Top-down more often. See Wikipedia article for more details.

Having to mock all the components before you can use them is tedious. And creating subgraphs before top-level graphs too.

What we need (while keeping the existing bottom-up way too) is to start with top-level boxes, add some inputs and outputs, connect the boxes and then decompose them in a similar way. We aren't ready to select/upload their components or even connect to a runtime yet - that is to be done later. But we would like to be able to import and export current graphs in JSON or FBP notation.

[forresto]

I've been thinking that the search dropdown will be more like a CLI, so there could be an option there for "new component."

1. Select new component
2. Edit name, inputs, outputs in modal
3. It generates CS stub code

This could be done without changing much under the hood, if keeping the CS step is OK.

The other piece that would be needed would be to convert/edit one of these stubs as a subgraph.

[trustmaster]

@forresto As a quick fix that would be acceptable in a short term and #241 exists for that. But in long-term we need to avoid CS stubs or at least hide them under the hood.

Let me give an example why. Imagine you're an engineer of enterprise-level systems; you know Java and Python and want to implement a new system in Java and Python. You know a little JavaScript and CoffeeScript to you is absolutely like Chineese is to a native American. A system is huge, so you start drawing it top-down: from topmost levels. In most cases you don't even know about all the ports ahead and if it's going to be a Component or a Graph. All you want to do at this point is to draw boxes, give them names, add some ports (with a GUI tool, not CoffeeScript because you can't get it right), connect the boxes, then decomposes some boxes into subgraphs, do the same with them, etc. For some boxes you might want to set component names, but there is no runtime connected or even existing yet - the graph comes first and the components are implemented in the runtime afterwards. When the runtime is up (implemented in Java + Python via sockets), you want to connect the graph to it, eventually.

So this one is more complicated and a longer-term goal, I think.

[forresto]

For the UI, I think that a special 'stub' type would make sense here. Then we could have

• edit inports
• edit outports
• edit as component
• edit as graph

in the context menu (or stub inspector).

What would be the expected ux if you decompose a stub component into a subgraph? What would the exports connect to? I suppose that they could just be hanging.

before after

[trustmaster]

@forresto My thoughts, exactly!

@jpaulm, @oleksandr What do you think?

[jpaulm]

@forresto, @trustmaster, @oleksandr A few comments based on my experience refining DrawFBP:

• your before diagram shows foo/ooo. I assume this came from the existing system, but at this point you just want a name - you aren't assigning this block to any library or repository - it might even turn out to be a human procedure... very high-level diagrams may well contain human-implemented steps, e.g. a travel agent booking planes, hotels, cars, etc.
• not sure what you mean by "before and after". IMO flowhub badly needs multiple windows - I want to be able to jump quickly between windows or even display 2 or 3 work items (network, subnets, components), and the error log, all potentially at the same time, dependent on screen size (in DrawFBP I use tabs, but now I think separate windows would have been better).
• I like your icons! Don't know if you have looked at JavaFBP, but in JavaFBP and C#FBP these icons actually become specific components in the subnet (SubIn and SubOut). I am not too clear on the case of ports in NoFlo, but the last note I got on the subject seemed to imply that port names are usually upper case, and are only converted to lower case for the JSON representation.
• just to make sure, you need to be able to take a node, assign a component to it, change it to a subnet (not necessarily preexisting), change it back to a component without losing the subnet (if you have started building one), and/or remove any implementation ideas from the node and start over! This is based exactly on what DrawFBP does - I think we did that part right!

So yes, this will be a big step forward! Looking forward to trying it out!

Regards,

Paul

[jonnor]

The stub components proposed could be persisted as .json "subgraphs" with only inports/exports properties (no nodes,connections). That would maybe require some changes to allow inports/expors which don't refer to any internal/private node/ports.

[trustmaster]

Bumping this one as it's still pretty relevant.

[bergie]

So, spent some time thinking about this yesterday. One approach would be to lean on the "specification" part of fbp-spec, where users could:

• Add "new component"
• Flowhub opens fbp-spec editor
• In the editor you can provide component name
• You can also provide port definitions (the specified interface of the component)
• ...and when you can, provide test cases

These "fbp-spec only" components would be available for usage in graphs, but obviously can't be run on a runtime before there is an implementation. For this, from the fbp-spec editor you could:

• Implement as graph
• Implement as code (submenu for supported languages)

And of course once implementation exists, one should be able to move between spec and implementation.

[jonnor]

What is the "fbp-spec editor"?

[bergie]

@jonnor an imaginary piece to be built to support this workflow

[jonnor]

I don't understand why this is tied to fbp-spec. fbp-spec just has a link to a component in form of a "topic" string, and I think it should stay that way. An "editor" for fbp-specs sould be for editing the actual testcases (add, remove, change their expected data).

A problem with these runtime-less fbp-spec-only components it that they can never be executed in a runtime, or meaningfully co-exist with real runtime components in a graph. So one has to build a 100% fantasy graph, then convert everything at once to a 100% real implementation.

I propose instead that we use subgraphs for "dummy components". This has the advantage that they can be combined with "real" components and executed in the runtime. And inside graphs one can use such components, or one can use real existing components, in combination. And gradually implement each of the stub graphs, either by wiring them up, or writing the code. One can of course have specs for such components/graphs too, nothing special about that.

There can be a command "convert to component", which could use the port information to fill in a template.

[trustmaster]

I agree with @jonnor that it should have nothing to do with fbp-spec. All that is to be known about these "prototype components" is their name, inports and outports. So there should be an UI to create/edit that prototype definition. And once the time comes to, there should be a way to complete that prototype into a functioning component or graph. The prototype itself can be a graph like @jonnor said, or an auto-filled component, or other JSON metadata that Flowhub can turn into a component or graph on request - whatever is more convenient. The key requirement is to be able to prototype components quickly, and then to proceed from prototypes to real code easily.

[trustmaster]

A minor thing: to speed up prototyping even more, it would be nice if the tool added IN and OUT ports as default, so that you could begin prototyping a graph by just adding components and wiring them to each other. And then enhance the prototype with more details on second take.

[jpaulm]

Hi Jon, not sure what the problem is here... In DrawFBP I can draw a diagram without filling in any component names - just short descriptions, and do a lot of design work quite comfortably at this level. I can at any point start filling in component names, or replacing nodes by subnets, but of course DrawFBP will not generate a network until all nodes have components or subnet names filled in.

And Vladimir, DrawFBP now automatically fills in OUT and IN port names on every arrow you draw... it's been doing that for several months... :-)

Regards,

Paul M.

On Tue, Jan 3, 2017 at 12:50 PM, Jon Nordby notifications@github.com wrote:

I don't understand why this is tied to fbp-spec. fbp-spec just has a link to a component in form of a "topic" string, and I think it should stay that way. An "editor" for fbp-specs sould be for editing the actual testcases (add, remove, change their expected data).

A problem with these runtime-less fbp-spec-only components it that they can never be executed in a runtime, or meaningfully co-exist with real runtime components in a graph. So one has to build a 100% fantasy graph, then convert everything at once to a 100% real implementation.

I propose instead that we use subgraphs for "dummy components". This has the advantage that they can be combined with "real" components and executed in the runtime. And inside graphs one can use such components, or one can use real existing components, in combination. And gradually implement each of the stub graphs, either by wiring them up, or writing the code. One can of course have specs for such components/graphs too, nothing special about that.

There can be a command "convert to component", which could use the port information to fill in a template.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/noflo/noflo-ui/issues/266#issuecomment-270176511, or mute the thread https://github.com/notifications/unsubscribe-auth/AATGJznbWM9b0vsHwpZumBN3_d0PsJXBks5rOoqBgaJpZM4B-6Oj .

[bergie]

@jpaulm the difference is that while DrawFBP is editing abstract graphs, NoFlo UI/Flowhub is usually used in connection with a live runtime, meaning that any graph changes cause changes to a running FBP network. And so if you suddenly add nodes that the network can't instantiate (since there is no code/graph backing them), you have weird errors.

[jpaulm]

Hi @bergie , it feels to me that you are missing the design phase! You say "NoFlo UI/Flowhub is usually used in connection with a live runtime", but how do you design a complex network where you don't know which components you will have to code and which can come off the shelf? Sounds like you (currently) need a different tool for that phase - but then you would have to switch tools halfway through the project. When you get a chance, could you walk us through the whole project lifecycle? I am probably missing something key... or maybe we're looking at the problem from different points of view...

[bergie]

@jpaulm well, yes. Hence this ticket ;-)

What we support is bottom-up design (start by building components, then connect them), but I totally agree with @trustmaster that supporting top-down design (start by defining graphs, then implement components as necessary) will be useful.

[jpaulm]

"Will be useful" seems a bit weak! :-) IMO it's absolutely essential - but of course it has to work in conjunction with bottom-up. In my book, I also describe two other "directions" (all of which are complementary): "start at the user's display requirements and work backwards", and what I call "centre out" - start with a core of processing, and then add frills like pretty-printing, editing, etc. And then of course we have to be able to migrate a prototype or a simulation to production without having to change design tools. Since you typically prototype areas of uncertainty, you shouldn't have to rewrite everything to get into production. I think any design tool must support all of these, as seamlessly as possible.

[jonnor]

As discussed with @bergie and @trustmaster here is the implementation plan for this.

the-graph

• [ ] Support for visualizing nodes differently, by passing in more styling information (as overrides)

Flowhub MVP

Stubs are a component-like thing, which is not sent to the runtime

• [ ] Support for adding a stub component, as a node in the graph

• [ ] Support for adding inport/outport on a stub, by passing a different context menus to the-graph for stubs

• [ ] Add persistence support for stubs. Similar to components, but stored separatly.

• [ ] Informing the user that graphs with stubs cannot execute on the runtime. Preferably with an informative message listing the stubs that need implementing. Bonus for pointing the user to how to do this.

• [ ] Support "implement as component" action on stub.

• [ ] Support "implement as graph" action on stub

Later/bonus

• [ ] Support for adding/editing an fbp-spec for a stub

• [ ] Support templating in the port information (names+types+description) for create

• [ ] Support "implement using existing component" on a stub

[bergie]

Just realised that the MsgFlo foreign participant registration format is roughly what we'd need for specifying non-implemented components

[bergie]

Current UI sketch for defining component interfaces: