universal dataflow graph editor?

[forresto]

I have been working on the Meemoo graph editor for a year or so, and I'm pretty happy with it. I have been thinking that it would benefit my project if I abstracted out all of the graph editing code from the Meemoo-specific code. Then I was thinking that all of the dataflow js projects could potentially benefit from working on a common editor.

Things that I would keep:

1. JSON graph format
2. Save to localStorage / gist
3. Selectable, draggable, droppable, resizable
4. Cut, copy, paste, with x,c,v keyboard shortcuts
5. Module library

Things to add:

1. Polishing touchscreen UX (lots of potential and room for improvement here)
2. Cleaning CSS, making it possible to look like Pure Data, Meemoo, or beyond using mostly CSS (+ some JS for SVG edge drawing)
3. Abstracting all data flow stuff from the views
4. Testing (:flushed:)
5. Abstract out save local/gist as optional plugins, to make it easy to extend to other services, implement auth, etc.

Questions:

1. Macros / subgraphs. How to represent this in the graph JSON?
2. Stick with Backbone.js?
3. Stick with jQueryUI?
4. Best interface to define how the different versions of source.connect(destination) is called from the GUI.
   • Named inputs/outputs

What do you think? Would this benefit your JS dataflow projects? Would you contribute to a common kickass graph editor?

@forresto - https://github.com/meemoo/iframework @sebpiq - https://github.com/sebpiq/WebPd @oampo - https://github.com/oampo/Audiolet @brianchirls - https://github.com/brianchirls/Seriously.js @dashersw - https://github.com/dashersw/pedalboard.js @cwilso - https://github.com/cwilso/WebAudio @bergie - https://github.com/bergie/noflo @idflood - https://github.com/idflood/ThreeNodes.js @daeken - https://github.com/daeken/Qtzweb

(Feel free to tag other related projects to this conversation.)

[sebpiq]

Sounds great to me !

What do you mean by "Save to localStorage / gist" ?

Things missing for me :

1. connections should be selectable as well
2. delete with "del"
3. not only should the whole thing be stylable, but you should be able to configure if connectors are horizontal or vertical.

What are you using JQuery UI for ? Backbone.js is great ... let's keep it.

[forresto]

I'm using Backbone.localStorage to save graphs to the browser's key/value localStorage. If you change something on a Meemoo app and click [app], [save local] it will be there for you in the future.

[save public] uses Github's Gist API as a quick way to share graphs publicly. This saves anonymous gists.

These functions should be plugins, to add or change as needed. I'll add that to the to-do list.

The ports could be positioned horizontally, vertical, or anywhere with CSS. The way that I see edges working is a function that takes (x1, y1, x2, y2) of the i/o ports and returns the svg path's d attribute, like M 767 25 L 50 31. This could be a straight line, curved, or fancy edge routing around modules.

I'm using jQueryUI's slider, button, selectable, draggable, droppable, and resizable. I guess it would be lots of work to replace it. TouchPunch got me part of the way to touchscreen usability, but I need to do more work there.

[kn0ll]

yes! i've been using audiolet quite a bit, but i've been avoiding UI graph related stuff because it's a pain in the ass. i realized meemoo was right for this- but the rest of the meemoo suite is inapplicable. abstracting the graph stuff out would be really helpful. assuming most graphs operated using the same API (add a package manager into the equation and now we're having fun...). my unqualified opinions in no order:

backbone is in every one of my projects, but to be completely honest, removing the backbone and jquery ui dependencies would be nice. if you're willing to support modern browsers exclusively, reimplementing slider/draggable etc. would not be much work since they all have native equivalents.

a note on inputs: if i understand correctly, we're talking about the horizontal/vertical justification of the inputs (along the edges of each node)? if so, horizontal/vertical would not account for absolute inputs, eg. the ims 20. (ignore the fact these are routing internal components- i would like to route between nodes this way as well).

a note on css: just a thought... would be nice to have minimal or no css required at all, and write the most basic styles inline through javascript. requiring css makes it slightly less portable.

[forresto]

Since this is a full-on graph building GUI, I'll shy away from the no-dependency ideal to start. But I'll make the HTML/CSS as minimal as possible.

Absolute-positioned ports for sure. (The Korg example will be tricky, as the SVG for the wires could block click events to the modules below. But I'm sure there is a workaround. WebModular puts everything in a canvas.)

This project will build on the ideas in https://github.com/h5bp/lazyweb-requests/issues/82 ... JSON graph formats are easy enough to translate between, but some version of {nodes: [], edges: []} seems sane.

The big question for me is how to define macros/subgraphs. We need some way to import somebody else's graph (an audio or visual effect) as a subgraph in our own. WebPD has recieve and send nodes as a way to publish inputs and outputs for a particular patch.

[forresto]

I should mention @idflood 's ThreeNodes. ThreeNodes is already fully-featured graph editor, honed to the task of building three.js scenes.

How could we include a ThreeNodes module in a Meemoo graph, defining ins and outs so it could accept textures or numbers from other modules?

[sebpiq]

Well ... actually, this subgraph problem exists in Pure Data, and is an open question in WebPD. I wouldn't use [receive] to handle that. The way I think I will do is actually make the idea of subgraph disappear ... if you just open the subgraph and make it part of the parent graph there's no problem anymore... which doesn't mean that you should display, and doesn't mean that you should forget there's a subgraph. ... just not handle connections differently. Ex : (I don't know the format you use in dataflow) :

{
    "graphs": [
        {"nodes": [1, 2, 3], "connections": [[1, 89], ...]},
        {"nodes": [89]} # -> a subgraph
    ]
}

[forresto]

I guess I'm confusing pd's send and receive with inlet and outlet. I think that any graph could define its default ins and outs, so that when it is included in another graph it automatically exposes those. You could then open that subgraph and change the variables that it is exposing.

[automata]

+1 so interesting to a Web Audio API node editor as well. Something like "blockly to nodes"?

[sebpiq]

[send] / [receive] are basically just named cables. In PD you can indeed define subgraph's inlets and outlets, which are basically its ins and outs. What about subgraphs being nodes ? Nodes have inlets and outlets right ? So if 'Graph' is just a subclass of 'Node', then that could work !?

@automata : WebPD will soon be a Web Audio API node editor :) I'm planning to rebuild it fully on Web Audio API. Some objects (Oscillators, filters, ...) will only add a very very thin layer on top of Web Audio API nodes.

[automata]

@sebpiq so let's work together because we are doing the same thing on Meemoo :-)

[sebpiq]

you mean that you plan on doing a web audio API editor for Meemoo ? Anyways ... now about the graph editor, I'm talking a bit for nothing ... I'd need to read the code first and then we can talk.

[automata]

yes! sorry about the OT... https://github.com/automata/iframework/tree/gh-pages/src/nodes (the audio*.js files) I think we can have WebPd as Meemoo main "audio engine" instead of rewriting everything from scratch.

[sebpiq]

Well ... that would be awesome I say ! One of the most important things to me with WebPD was to build it as a DSP library, so there shouldn't be any problem for using it as an audio engine (at least no more problem than with any other JavaScript DSP library).

[forresto]

It makes sense to think of "graph" as a subclass of "node." In this case, if you import two instances of graph B into graph A and then edit one, the other won't edit. Sometimes this is what you want, and sometimes you want the two instances to be the same.

Maybe it could work both ways...

Unsynced:

{
  nodes: [
    {id: "b",  graph: {nodes:[], edges:[]}},
    {id: "b2", graph: {nodes:[], edges:[]}}
  ],
  edges: []
}

Synced:

{
  nodes: [
    {id: "b",  subgraph: "b"},
    {id: "b2", subgraph: "b"}
  ],
  edges: [],
  subgraphs: [
    {id:"b", nodes:[], edges:[]},
  ]
}

Then the question would be how to represent the difference in adding/editing the different kinds of subgraphs in the UX.

[sebpiq]

Why not something like :

{
    root: {
        type: "graph",
        nodes: [
            {id: "b",  type: "graph", nodes: [], edges: []},// normal subgraph
            {id: "b2", type: "abstraction", template: "mySubGraph"},// Subgraphs synced to the same template
            {id: "b3", type: "abstraction", template: "mySubGraph"}
        ],
        edges: []
    },
    templates: [
        {id: "mySubGraph", nodes: [], edges: []}
    ]
}

[forresto]

"template" seems good to me. I don't think we need to separate the "root" graph out like that, what the advantage be?

[sebpiq]

Perfectionism ? Fascism ?

[sebpiq]

No ... seriously that way the root graph is just a graph like any other, same attributes and so on. So then you leave the top namespace free for putting other stuff (like "templates"), which don't really belong to the graph itself. ... and it is just cleaner.

[brianchirls]

The is a very cool idea, and I'd definitely be interested. For it to work with Seriously.js, it would have to have a lot of hooks in and out with pretty fine-grained control. I would probably want to build some kind of wrapper between the graph UI and a Seriously node graph. And would want to be able to customize what goes in the view for each node. For example, I'd like to be able to put a thumbnail canvas to preview output for each node.

It'd be helpful to have good controls for inputs of different types, or at least the ability to build them and plug 'em in. For example:

• Number (with or without optional min/max/step)
• Color (RGBA, HSLA)
• Enum
• Boolean
• vector (2d, 3d, etc)

[sebpiq]

@brianchirls : I might be wrong ... but the way I see it, the graph editor would be completely naive, in the sense that it doesn't care what actually goes through the wires.

[bergie]

Sounds good! I've written some of my thoughts about the requirements for a flow-based programming editor in bergie/noflo#1

For what it is worth, here is the NoFlo JSON format: https://github.com/bergie/noflo/blob/master/examples/linecount/count.json

Corresponding "FBP notation" file in https://github.com/bergie/noflo/blob/master/examples/linecount/count.fbp

[brianchirls]

That sounds reasonably cool. Have you thought much about event triggers for modifications? I'm thinking...

• new node
• deleted node
• node connected
• node disconnected
• input modified (maybe node connected/disconnected is included in this one)

Each trigger should include only the data that was modified so you don't have to go traverse the whole graph again to look for changes.

[bergie]

@brianchirls you could go even further, and do eventing for things like:

• node crashed
• packet transmitted in a connection

Especially for the packets a "step debugger" mode would be useful where you would see and confirm each packet before it gets received by an inport.

[sebpiq]

packet transmitted in a connection

yep ... I need that as well.

[brianchirls]

@bergie That's interesting. But I would think most or all of these frameworks handle the packet transmission internally.

If you think of it as an MVC, then (Seriously.js|WebPD|ThreeNode.js|etc) is the Model and dataflow is the C. V is both dataflow and the canvas, speakers, etc.

But I could see where it'd be helpful to allow the model to tell dataflow when it's passing packets and allow an outside listener to be alerted when that happens. Or is that what you're getting at?

[brianchirls]

By the way, you all have insanely cool projects.

[bergie]

@brianchirls yep, not actually touching how data flows inside a FBP framework, but instead hooking to it using some sort of observer-kind of API.

[sebpiq]

Yep ... same here ! I'd need to update the visual of a node when data goes through. For example blinking light. But I think this is part of the bigger problem of "how is a dataflow node (node view) tied to an actual Computing node (model)" ?

[sebpiq]

Ohh !!! yeah ! Backbone right ?

[sebpiq]

and @brianchirls : you meant "we" all have ... right ? :)

[dashersw]

Check out pedalboard.js's UI component system. I have a long history of programming reusable components and I care a lot about browser performance issues. My two cents would go to such a component system - it's perfect for separating model and view logic, incorporating some form of MVVM pattern (without the fancy data binding lunacy). It carries its template markup within, though I've always thought of CSS a finer detail not to be handled within the components.

Personally, I'd hate to use a jQuery / Backbone.js solution.

[bergie]

Backbone has the major advantage that there is quite a large community that actually understands it. But I'm biased, as I've written some big tools like VIE and Create.is on it

[brianchirls]

@sebpiq +1

@dashersw I find the data binding lunacy thoroughly annoying, and probably inappropriate for this project. I'm trying to render a dozen canvases at 60fps here. So agreed.

Actually I have a kinda-neat "micro template" tool I threw together and have been wanting to open source. Instead of parsing html-esque template code and filling in values, you give it a JSON blob that describes a DOM sub-tree, and it returns an object with named properties that point to elements within the resulting tree. And you do whatever you want with those elements. The whole thing is about 50 lines of code. I haven't jsperf-ed it yet, but it should in theory be pretty quick 'cause there's no string tomfoolery. Might be useful here.

[dashersw]

@brianchirls I'd love to see your templating solution and try it out on a performance ride on heavy DOM creation.

[brianchirls]

@dashersw Excellent. Lemme get it into a repo. Gotta run it by some people first though 'cause it's from a different project.

[sebpiq]

Personally, I'd hate to use a jQuery / Backbone.js solution.

Yeah ! Let's reinvent the wheel :) No seriously, why not ? Those are great libraries ... and can save quite a bit of work (and time). About the node rendering, I don't really see why would you need to re-render often ? What I tend to do those days (I don't know much about rendering performances, so that might be a stupid solution), is to do stuff like that :

<div id="myElem">
    <div class="stuffToShowInState1"></div>
    <div class="stuffToShowInState1"></div>
    <div class="stuffToShowInState2"></div>
</div>

<style>
    .stuffToShowInState2, .stuffToShowInState1 {
        display: none;
    }
    #myElem.state1 {
        background-color: pink; /* why not ?*/
    }
    #myElem.state1 .stuffToShowInState1 {
        display: block;
    }
    #myElem.state2 .stuffToShowInState2 {
        display: block;
    }
</style>

so basically I render all the states of the element, and just switch its class. Of course this is limited to state changes and there is cases where you actually need to re-render ... but still, I have noticed that in many cases, changes in the model can be just seen as state change.

[forresto]

So then you leave the top namespace free for putting other stuff (like "templates"), which don't really belong to the graph itself.

I think that they do belong to the graph. If you were to copy that root graph you would have to traverse up to find "mySubGraph," so it seems like more work than having templates on the same level as nodes and edges. I vote for making the root graph the root of the {}.

But the graph format doesn't have to be standardized, since we will have to translate the JSON to our own addNode() and connect() calls.

It'd be helpful to have good controls for inputs of different types, or at least the ability to build them and plug 'em in.

Yes, the editor doesn't care what goes through the wires, but some common UI elements like color pickers and sliders for manually inputting values would be good.

browser performance issues

I have been using browser profiling tools with Meemoo, but I'd be interested if there are any obvious issues with this benchmark that could be faster without bb/jqui. (It is a fractal drawn with recursive image transforms.) The only thing that I notice is that the framerate dips when dragging nodes because of the SVG edge redrawing.

[sebpiq]

I think that they do belong to the graph.

In Pd, the way it works : there can be several root graphs loaded at the same time. When you modify one "templated" subgraph it is modified in all the suggraphes of all the root graphes actually running. Basically "templates" is like a global containing all importable objects. So I think it belongs to the application (i.e. "root") level, and not to a particular graph. And also, just for the sake of consistency and cleanliness, there should be always same data in a graph, it doesn't matter whether it is a root or not. A graph should be just a graph.

[forresto]

@sebpiq, OK, I'm with you. I'll work on normal (inline) subgraphs first.

@ All, I started work the foundations of the data structure and just pushed it. I'm starting from scratch in order to reduce boilerplate and spaghetti as much as possible.

In Meemoo I'm using yepnope to load node code and libraries as needed. Is that feature worth putting in this? Would a heavier module loader be overkill?

[sebpiq]

In Meemoo I'm using yepnope to load node code

I don't really understand why this would be needed. If this is needed in dataflow itself then I'd say yes, if this is just to offer more features I'd say no ... let's keep it simple and focused.

This week I have lots of stuff to do, but starting next week, I'll start to take a look and implement some stuff.

[bergie]

NoFlo's UI code has been now split out from the main repository, and is in https://github.com/bergie/noflo-ui

I'm still using jsPlumb for now, but wondering when dataflow would be in a shape where integrating it would make sense.

[subtleGradient]

Ahoy all. I'm working on Design Tools at Facebook. We use Quartz Composer a lot, but it doesn't really have a future. I'm looking at various FBP-esque runtimes like Vuo.org and NoFlo. There are a ton of different implementations. Meemoo is really awesome!

Some of the tools we are building will use the WebKit Inspector Protocol. Cf. PonyDebugger. It woul be epic if we could collaborate on integrating a visual programming tool (or many!) with everything that speaks that protocol. This would allow us to use multiple editors with multiple runtimes. Multiple people editing the same graphs simultaneously from different machines would be easy. It's all just JSON RPC over web sockets with a standardized API.

I'm in the middle of building a web inspector protocol server for photoshop. I hope to add web inspector support to Quartz Composer next.

If anyone is in the Bay Area, please drop by the Menlo Park Facebook office for lunch.

[subtleGradient]

One thing that may help us collaborate would be to come up with a standard definition syntax for graphs and components like CommonJS did with the package.json standard.

For Graphs there's the FBP format, NoFlo JSON format, Meemoo JSON format and Quartz Composer QTZ binary PLIST format. But what about individual components? Quartz Composer QTZ files can double as components when placed in your patch library folder. What other formats are available for describing an individual component and its in ports and out ports?

[automata]

@subtleGradient thank you for the points, NoFlo and vuo.org are really interesting. We are looking for me way to make real time collaboration possible on Meemoo and JSON RPC over web sockets sounds so good. We were looking to use share.js JSON OT to make Meemoo graphs shareable between multiple users.

[forresto]

@subtleGradient I like the idea of a common graph/module format, but there are challenges...

The libraries that I have explored (Seriously.js, Web Audio API, Tuna, Meemoo) all have a pretty similar format; modules have arrays of inputs and outputs that can have information about their type, min, max, default, etc. It was pretty easy to parse Tuna's effect modules to work in Dataflow. http://forresto.github.com/dataflow-webaudio/

One issue with making something standard is that there are different systems that control how the data flows. Dataflow just passes out.connect(in) to the Web Audio API when the wires connect. Meemoo modules hold variables and canvases and process/pass them with a requestAnimationFrame tick.

As long as the libraries have differing flow systems there is going to have to be some wrapper code to translate how they communicate.

[subtleGradient]

I'm sure implementations will differ greatly. I'm more interested in the component and graph formats for building implementation agnostic tools. e.g. When connecting a noodle, the tool fires the generic Connection.added event (or whatever) with the details about each node and their ports. It would be up to a specific implementation to listen for that event and take whatever implementation specific action is required to actually make it happen. But the tool shouldn't have to care about it.

[subtleGradient]

ShareJS looks incredibly epic! (I had seen it before, but forgot all about it) If there were a common document format for a graph, you could setup a ShareJS server for your graphs and then multiple runtimes and editors could connect as clients. Maybe even hook different runtimes to a different subtree of the graph. Very interesting indeed! I'll have to think about how to use this…

[bergie]

I wrote some thoughts and requirements we have for the NoFlo editor in Thinking about the flow-based programming user interface.

The current plan is to build the editor to a usable state still during this winter. I hope the dataflow code base will be in the state where we can rebase from our custom jsPlumb implementation to that.

@subtleGradient shame that you wrote these comments just now! I was still in the Bay Area working on some NoFlo things a week ago, but now I'm back in Berlin. But there will be other trips...

[bergie]

@subtleGradient @forresto here is an interesting post showing the kind of prototyping scenarios we'll eventually need to support: https://news.layervault.com/stories/3117-facebook-home-prototyped-in-quartz-composer--tutorial

Now since bergie/noflo#63 the NoFlo engine also runs in the browser, so I can also look at building interaction components.

[subtleGradient]

Now you know what I've been up to. At Facebook, we have a lot of custom in-house QC patches that made prototyping Facebook Home and Chat Heads a lot easier. It'd be awesome to get all our stuff running in the browser and Node.