Add support for the JSON graph definition format

[bergie]

It would be great if goflow could implement the same JSON format for defining graphs as we use in NoFlo. This would eventually help with utilizing tools like the UI across both systems.

[trustmaster]

Sure, that's one of the top priorities. Since Go is a statically typed compiled language I think of a tool that would convert NoFlo JSON or FBP DSL network definitions into plain Go code which is then compiled and run.

[bergie]

@trustmaster that sounds like an interesting approach. With NoFlo we use the JSON file for constructing the network at runtime... this means that the components are loaded from code, but the connections between them are build at runtime.

The benefit of this is that we can also modify the network while it is running.

[trustmaster]

@bergie Yes, constructing networks at runtime is relatively easy with dynamic languages. But in Go, despite its amazing (for something that compiles to a binary) reflection capabilities, it is impossible to create an object of arbitrary type represented with just a string name at runtime. Even if that was possible, it would be tricky to create instances of classes defined in packages which weren't even compiled into the binary. So I think compile-time is the only option for Go currently.

Is it possible to do network reconstruction at run-time in Java? This might be an important question since Java is in NoFlo's stretch goals.

[ghost]

@trustmaster: It shouldn't be any problem to rewire a graph network of components in runtime in any language, if the application just recognize a configuration file (as in this case, in JSON format) and then picks the corresponding component(s) and connects them according to the info found in the config file.

What it takes though, is of course that the components already exist (before loading the config file) and that the main (GoFlow) application is designed to act as a "framework" supporting this kind of runtime-wiring. Using pointers to assign In/out ports is runtime stuff.

What's new to FBP is to design the components generically, according to the FBP concept, while the actual assignments of the components may use whatever "traditional" connection mechanisms, preferably the mechanism should work also if distributed over physically separated machines. I think there's a library for Go which supports distributed channels etc.

BTW, for several years ago I designed a huge ERP system, with Delphi/Pascal, where the entire meta system, including state machines (or "behavior machines" as I called it) is wired in RT, according to a hand written xml-file. It could load all the business logic in runtime without having to restart the application kernel/server. That is, the application could rewire & redefine everything state-related in RT. I even (remotely) created "repair-states" for invalid data, such as unassigned object links etc for transport vehicles in the system, by defining a "garage-state" for which I added some simple scripts to be executed on transition. In so doing I could repair runtime data - from a location in another country - while users watched it all happening one the screens (I of course also made sure to transit the involved repair-objects into a temporary "lock-state" while doing all the dirty repair work). After the repair was done, I deleted the definitions from the config.xml -file and reloaded it, and so left no traces after me of what had taken place.

From the user's perspective a (supernatural) "wonder" had taken place. :)

A "renown major car maker in Sweden" uses that system since last August.

So yes, in a statically compiled language you can do exactly what you want to do, if you design the framework for supporting such "dynamic" stuff. And a FBP framework is essentially about loading, wiring, and executing networks (graphs) of existing components. Which is what any meta system also is (or can be).

Regards from Sweden (not Finland), // Rolf

[trustmaster]

@RIL-

That's really interesting. How did you manage to create new instances of state machines at RT? For example, given this in config file:

"processes": {
  "Split by Lines": {
    "component": "SplitStr"
  }
}

How do you create an object of class SplitStr given just the name of the class as string in Delphi?

[ghost]

@trustmaster: I would suggest registering components in a separate GoFlow - component library, and have the library to return a FBP component by (string) name.

That is, comp = GoFlowCompLib.GetComponent("SplitStr"); // :) comp.in = prevComp.out;

The interface of a component would, well, be known, (it's assignable in/out-connectors) so it only needs to be assigned to other components by the framework.

In Go which has reflection this should be even easier to achieve than in the system I designed (I used Delphi6 at the time of the basic system design, so I didn't have access to reflection, which was introduced in later versions).

// Rolf

[trustmaster]

@RIL-

That looks like a very high-level factory/wrapper, while in GoFlow we currently try to keep things as close to bare metal as possible.

Go's reflect package doesn't let you create objects given just a type name string. Though it might be possible to instantiate arbitrary components at run-time if they exported their reflect.Type in a hash table. This concept needs to be proved. Thanks for pointing it out!

For now I'd go for a static graph code generator because of the following:

• It produces less footprint at RT using mostly native data types.
• It benefits from compile-time checks.
• It is simple and less error prone.

But in long-term if we want to achieve run-time evolution of long-running applications it will require reflection-based instantiation and network reconfiguration. That is all useful until you need to add a new component or update the code of an existing one, then recompiling and restarting the application is a must (given that Go is statically linked and there are no plans for dynamic linking).

[ghost]

@trustmaster:

Assignments as "ComponentA.out.ComponentB" is in effect using pointers. Can't get much closer to the metal than that. At execution time that means there's no overhead whatsoever, exept for the initialization of the graph of course.

The fact that Go is "close to the metal" doesn't mean that one cannot do anything with it that is on a higher (conceptual) level than the language itself. Instead it only means that Go is a perfect language for designing such (frame work) systems as FBP. That goes for Delphi as well. :)

// Rolf

[trustmaster]

@RIL-

I'm not worried about connections, that is already done at run-time in GoFlow using reflection. I'm worried about process instantiation and binary linking (e.g. Go compiler doesn't allow unused variables and packages in the code).

[ghost]

A register/factory (package) can return instantiated component objects by (string) name.

I also plan to generate Go code directly from model. But of course it would be most desirable to support a generic graph format enabling runtime configurations since that's what's Business Enterprise systems is about theses days (you can't really restart the heart of an enterprise business, that is application servers, while hundreds or thousands of employees are logged in...).

As for entirely new components it's not a big deal. In those cases, just wait until the late hours to rebuild and restart the system.

== COMBINE THINGS ==

What I was also thinking about was that @bergie's take on FBP with NoFlow could be helpful for cases where there's need for a new component in runtime. Just make a generic NoFlow component, which internally knows all about how to invoke node.js code to get its job done.

For this reason, among others, the components really should be identified by an internal string name property. that is, any NoFlowComponent will be exactly the same "class" (struct) in Go. Only its name will differ. All the configuration of such an object would be stored in a JSON object on disk, and the unique NoFlow object initialized in the Go-NoFlowComponent at runtime. And it would do what it does according to its component name.

== NOT EITHER OR ==

Conclusion: Combining statically compiled/linked programs with dynamical ones is optimal for real world enterprise business systems development. See the example above which I already mentioned. There's actuallly no need for reflection at all.

@Bergie's experience with NoFlow could prove to be most valuable if combined with Go. Think "NoSQL". It means not "no SQL", it means, "noth either or SQL, but both". Same goes for static and dynamic languages. Because that's what we've always been doing (we've used config files). :)

// Rolf

[ghost]

@bergie wrote "It would be great if goflow could implement the same JSON format for defining graphs as we use in NoFlo. This would eventually help with utilizing tools like the UI across both systems."

As a matter of fact, this (Js GUIs) is exactly what the "big" system needs which I designed. Developing user interfaces is a nightmare, it takes time, and it costs money. GUIs really would benefit from using NoFlo FBP technology. The application server OTOH may still make best use of static languages like Go/GoFlow.

What's needed is, apart from the flow model itself, a GUI design tool which directly generates NoFlo graphs from the graphical user interface mockup.

Optionally with all the wiring and references to the backend logic serving the UI components, for example with RESTful data. Well, in version 2.0.

Go-Servers with Js-GUI-clients - a potentially perfect combination.

// Rolf

[edited]

[trustmaster]

@RIL-

Your messages actually gave me an idea on how to implement a run-time process factory in GoFlow using pointers to constructor functions or reflection in the worst case. I'll give it a try.

[ghost]

@trustmaster , Sounds interesting!

Information Packets - Just want to point out that in order to get the best out of the FBP concept, IP's and IIP's are crucial in that they provide with super fast "indirections" (pointers or references) for data while also preventing data races.

I've used IP-like concepts when processing huge amounts of raw data since they can be reused and so avoiding repetitive memory allocations. In demanding real world systems such is gold. IP simplifies things.

Keep us informed. And don't hesitate to drop a mail. I'm out for FBP, preferably using Go for backends.

// Rolf

[bergie]

Related to the JSON format, the other thing of potential interest is the upcoming NoFlo protocol that would allow the UI to talk live to different FBP systems. See noflo/noflo#107

[trustmaster]

@bergie Sounds good, I was thinking of using ZeroMQ for interprocess and remote communication too. Let's use the same port-to-port protocol there.

[bergie]

@trustmaster ZeroMQ is one of the transports I had in mind as well. Please add your feedback to the NoFlo protocol issue, so we can ensure things work.

My current testbed for the protocol is simply running NoFlo in a separate IFRAME and passing messages via window.postMessage. But that is easily translatable to other transports as well.

Tests for port-to-port protocol: https://github.com/noflo/noflo-runtime-iframe/blob/master/spec/component.coffee Tests for the other parts of the protocol (graph, network, and component): https://github.com/noflo/noflo-runtime-iframe/blob/master/spec/network.coffee

[trustmaster]

@bergie Is it possible to use NoFlo-UI as a simple JSON editor adding components manually or does it require implementing a complete NoFlo runtime, supporting NoFlo protocol and component library?

[bergie]

@trustmaster right now it requires the complete protocol. Of course you could "fake it" by implementing your on runtime plugin

[trustmaster]

@bergie OK, NoFlo protocol implementation and UI plugin will be the next step. Regarding this issue, I have made a few extensions to the JSON format, so some more details about the graph can be specified.

First one is setting custom connection buffer size (optional, defaults to 0 in both NoFlo and GoFlow), e.g.

{
    "src": {
        "process": "starter",
        "port": "Out"
    },
    "tgt": {
        "process": "generator",
        "port": "Num"
    },
    "buffer": 20
},

The second one is used to switch processes between Sync and Async mode (GoFlow supports both, currently Async is default) by specifying "sync" flag in processes (optional, default depends on runtime):

"generator": {
    "component": "sequenceGenerator",
    "sync": true
},

What are the chances for supporting these in NoFlo UI? Is there a better place to work out the JSON graph definition spec and update it in future? I've added an issue in noflo/noflo.github.io#42, but that's probably a wrong place.

[bergie]

@trustmaster looks good, though I would use the metadata key for storing both of those. So:

{
    "src": {
        "process": "starter",
        "port": "Out"
    },
    "tgt": {
        "process": "generator",
        "port": "Num"
    },
    "metadata": {
      "buffer": 20
   }
},

and

"generator": {
    "component": "sequenceGenerator",
    "metadata": {
      "sync": true
    }
},

The metadata key is where NoFlo UI also stores things like routes (wire colors), node labels (in case they're different from ID) and node coordinates. See for example https://gist.github.com/bergie/6733886

The buffer and sync keys might be interesting also for NoFlo when moving forward with noflo/noflo#109

[trustmaster]

@bergie What's the benefit of using metadata over direct fields? I thought metadata is UI-specific properties which runtimes normally ignore, while the rest is more related to the graph functioning itself.

[bergie]

@trustmaster the main benefit is that NoFlo keeps the metadata fields intact, while otherwise it only reads/writes keys that are part of the spec. So right now if you were to load and save a goflow JSON file through NoFlo UI, those additional keys would get nuked.

Besides, I think things like buffer sizes and sync vs async are, in fact, graph metadata :-)

[trustmaster]

@bergie I'd say that's an arguable statement, but it doesn't make much difference to me as long as it is very easy to refactor whenever necessary. Let's stick with metadata for now then.

Still, would be nice to update the spec page one day as it doesn't even have info about exports, not to mention any kind of explanations :)

[trustmaster]

Switched to metadata in f996dfc. Closing this issue for now, next stop is NoFlo protocol.