jsfbp that doesn't use fibers?

[tlrobinson]

Fibers are neat, but the majority of node.js developers don't and won't use them, and they don't work in browsers at all. It would be nice if we could provide a "classical FBP" system that works using standard JavaScript APIs.

Actually, after reading http://www.jpaulmorrison.com/fbp/noflo.html I'm pretty confused about how JSFBP is considered more "classical" than NoFlo. That page suggests the key difference is that NoFlow is single-threaded (because JavaScript is single-threaded) and thus not suitable for CPU-intensive programs, but then admits JSFBP is also single-threaded. So is the main difference actually the synchronous vs asynchronous style of programming?

Selective reading of ports and backpressure can both be accomplished in an asynchronous style. For example, Node-style Streams have pause and resume methods for handling backpressure. The port could buffer up to a fixed number of IPs then call pause on the stream when the buffer fills up. The port could have a read method that takes a callback (or returns a promise) which gets called when the next IP is available, and calls resume on the stream if the buffer is no longer.

To take advantage of multiple cores you could use something like WebWorkers, though I think a generic system based on a standard IPC mechanism (e.x. ZeroMQ) and serialization protocol (e.x. Protocol Buffers) would be even cooler, allowing you to write components in any language, or even distribute processes across machines.

But anyway, there are several options for sync and async styles in JavaScript:

Asynchronous style:

1. EventEmitters or Streams (see above)
2. raw callbacks: probably Node-style callback(err, result) callbacks. This is my least favorite option, but is still the most popular style in Node.js. I probably wouldn't bother with this.
3. promises: promises are gaining a lot of traction in the JavaScript world lately, but they're still a very asynchronous style

Synchronous style:

1. generators (ES6, can be transpiled to ES5): not available in browsers except via transpiling
2. await (ES7, can be transpiled to ES5): lets you write synchronous style code that awaits on promises
3. some other synchronous-style transpiler: e.x. streamline.js

Also this may be useful relevant reading: https://github.com/kriskowal/gtor

EDIT: task.js is another possibility: http://taskjs.org/

[tlrobinson]

Also, I would argue that this is incorrect:

JSFBP only requires the addition of a small package on top of node.js, node-fibers, which is easy to install. Even though it does not use pure JavaScript, neither does node.js, but the latter is becoming more and more accepted.

(http://www.jpaulmorrison.com/fbp/software.html#JSFBP)

A full Node.js environment could be emulated in the browser (and indeed "browserify" attempts to do this), but node-fibers can't be easily emulated in the browser because it introduces "blocking" semantics (something similar can be implemented once ES6 generators are available, though: https://stackoverflow.com/questions/18293563/can-node-fibers-be-implemented-using-es6-generators)

[alfa256]

Honestly I can't see how an API that looks like JavaFBP and other classical implementations could be made with traditional Javascript. If you can find a way please tell.

This kind of api:

while( (a = read("in")) !== null ){
   send( "out", a.value + 5 )
}

Blocking is not a problem but a feature, and I'm not really convinced that generators can emulate this behavior. I hope it does though, I simply couldn't find a reasonable way.

[tlrobinson]

Here's a simple example (using the Bluebird promise library and ES6 generators, available in iojs and recent versions of node with the "--harmony" flag)

(obviously read and send are fake versions of the real thing, but it should give you an idea of what's possible)

var Promise = require("bluebird");

var messages = [1, 2, 3];
function read(port) {
  return Promise.delay(1000).then(function() {
    if (messages.length > 0) {
      return { value: messages.shift() }
    } else {
      return null;
    }
  });
}

function send(port, value) {
  console.log("sending to port '" + port + "': " + value)
}

add5 = Promise.coroutine(function *() {
  var a;
  while( (a = yield read("in")) !== null ){
    send( "out", a.value + 5 )
  }
});

add5()

[jpaulm]

Hi @tlrobinson, I wrote http://www.jpaulmorrison.com/fbp/noflo.html before I got the idea of developing JSFBP, so it's not surprising that it doesn't reflect my thinking about JSFBP. At the time I didn't think it was possible to build a "classical" FBP system using JS, and NoFlo is certainly very different from classical FBP. It seems to have a definite niche, but it's a different one from that of classical FBP.

As you may be aware, I have been working with classical FBP for 40+ years, and much of that time it was being used to run most of the batch code for a major bank (using a green thread "classical" FBP implementation). So I believe I understand it quite well, and I have a lot of faith in it.

Around Christmas, my son suggested I look at return and yield, and, while researching that, I came across a reference to Marcel Laverdet's node-fibers in @bjouhier's excellent blog, Bruno's Ramblings - https://bjouhier.wordpress.com/ . In it he compares a number of JS approaches to getting rid of "callback hell", and the thing that stood out to me about node-fibers is that it supports multiple stacks, which in my experience is key to building a "classical" FBP implementation.

I did not see another approach that was as promising, and in fact, using node-fibers, it took me no more than a few days to come up with a very promising proof of concept, which I have been developing since. Your feedback and @comfreek's have been very useful as my JavaScript skills are pretty rudimentary!

I agree that node-fibers is a bit further away from plain JS than nodejs, say, but to me it still looks like the most promising technology for taking advantage of JS. We have discussed worker threads, and the consensus is that the overhead is too high and communication between worker threads too "narrow-band". I have also been warned to avoid Promises!

If people are willing to add nodejs to basic Javascript, I fail to see why they would be so unwilling to add one more component - namely node-fibers. You raise a good point about not being able to browserify node-fibers, but maybe there are other solutions to getting JSFBP in the browser - however, I see no immediate urgency to solve this problem. I will be quite happy if JSFBP can supports servers, providing good performance and good maintainability. This at least gives users a single language on both client and server - even if different mindsets!

We have used back pressure, selective receive, and information packets with defined lifetimes and ownership as sort of a litmus test for classical FBP

• if you can provide these functions with the tools you mention, and decent performance and maintainability, I would be very interested in seeing your solutions. A number of people have come up with FBP-like approaches over the last few decades, and I'm looking forward to seeing how these all work out, as we gain a deeper understanding of this technology. It might be timely to try to come up with a checklist of success factors, so we can measure the different offerings against it. I can tell you one thing, though: once you have made the paradigm shift described in my book, you can't go back! And I would venture to say that, if you can go back, you haven't made the paradigm shift! :-) That's 40+ years of experience talking!

Best regards, and many thanks,

Paul M.

On Feb 11, 2015 5:40 PM, "Tom Robinson" notifications@github.com wrote:

Fibers are neat, but the majority of node.js developers don't and won't use them, and they don't work in browsers at all. It would be nice if we could provide a "classical FBP" system that works using standard JavaScript APIs.

Actually, after reading http://www.jpaulmorrison.com/fbp/noflo.html I'm pretty confused about how JSFBP is considered more "classical" than NoFlo. That page suggests the key difference is that NoFlow is single-threaded (because JavaScript is single-threaded) and thus not suitable for CPU-intensive programs, but then admits JSFBP is also single-threaded. So is the main difference actually the synchronous vs asynchronous style of programming?

Selective reading of ports and backpressure can both be accomplished in an asynchronous style. For example, Node-style Streams have pause and resume methods for handling backpressure. The port could buffer up to a fixed number of IPs then call pause on the stream when the buffer fills up. The port could have a read method that takes a callback (or returns a promise) which gets called when the next IP is available, and calls resume on the stream if the buffer is no longer.

To take advantage of multiple cores you could use something like WebWorkers, though I think a generic system based on a standard IPC mechanism (e.x. ZeroMQ) and serialization protocol (e.x. Protocol Buffers) would be even cooler, allowing you to write components in any language, or even distribute processes across machines.

But anyway, there are several options for sync and async styles in JavaScript:

Asynchronous style:

1. EventEmitters or Streams (see above)
2. raw callbacks: probably Node-style callback(err, result) callbacks. This is my least favorite option, but is still the most popular style in Node.js. I probably wouldn't bother with this.
3. promises: promises are gaining a lot of traction in the JavaScript world lately, but they're still a very asynchronous style

Synchronous style:

1. generators (ES6, can be transpiled to ES5): not available in browsers except via transpiling
2. await (ES7, can be transpiled to ES5): lets you write synchronous style code that awaits on promises
3. some other synchronous-style transpiler: e.x. streamline.js

Also this may be useful relevant reading: https://github.com/kriskowal/gtor

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14.

[ComFreek]

That's why I suggested to use an abstraction layer in issue #13. Such an abstraction layer would ideally allow us to switch between different implementations, e.g. one for Node.js and another one for browsers. Another big and related advantage is that the implementation for browsers could only emulate Fibers, so that jsfbp is ready to run in browsers even though not with full performance.

[tlrobinson]

If people are willing to add nodejs to basic Javascript, I fail to see why they would be so unwilling to add one more component - namely node-fibers

This is simply not going to happen. AFAIK there has been zero serious discussion of adding fibers to the browser, and even if that started happening it would be many years before they were implemented and widely available.

Compare that to ES6's generators, which are built into current versions of Node.js, in the next version of JavaScript, so likely to start showing up in browsers soon. As you can see from my example above, you can get you something pretty similar to fibers. It's possible there's some reason why you can't implement classical FBP with generators instead of fibers, but I don't know yet (note that Bruno also mentioned generators and preprocessors like Streamline.js as a solution to callback hell)

[alfa256]

Could you try making a prototype that achieves a functionality equivalent to Collate ?

2015-02-12 13:26 GMT-03:00 Tom Robinson notifications@github.com:

If people are willing to add nodejs to basic Javascript, I fail to see why they would be so unwilling to add one more component - namely node-fibers

This is simply not going to happen. AFAIK there has been zero serious discussion of adding fibers to the browser, and even if that started happening it would be many years before they were implemented and widely available.

Compare that to ES6's generators, which are built into current versions of Node.js, in the next version of JavaScript, so likely to start showing up in browsers soon. As you can see from my example above, you can get you something pretty similar to fibers. It's possible there's some reason why you can't implement classical FBP with generators instead of fibers, but I don't know yet (note that Bruno also mentioned generators and preprocessors like Streamline.js as a solution to callback hell)

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-74101216.

[jpaulm]

If you can find a way to simulate multiple stacks in JS, then we should be able to do something! On Feb 12, 2015 11:59 AM, "alfa256" notifications@github.com wrote:

Could you try making a prototype that achieves a functionality equivalent to Collate ?

2015-02-12 13:26 GMT-03:00 Tom Robinson notifications@github.com:

If people are willing to add nodejs to basic Javascript, I fail to see why they would be so unwilling to add one more component - namely node-fibers

This is simply not going to happen. AFAIK there has been zero serious discussion of adding fibers to the browser, and even if that started happening it would be many years before they were implemented and widely available.

Compare that to ES6's generators, which are built into current versions of Node.js, in the next version of JavaScript, so likely to start showing up in browsers soon. As you can see from my example above, you can get you something pretty similar to fibers. It's possible there's some reason why you can't implement classical FBP with generators instead of fibers, but I don't know yet (note that Bruno also mentioned generators and preprocessors like Streamline.js as a solution to callback hell)

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-74101216.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-74108180.

[tlrobinson]

Can Collate be written in the current version of JSFBP (using fibers)?

[jpaulm]

Absolutely! The structure and even coding would be very similar to https://github.com/jpaulm/jsfbp/blob/master/components/rrmerge.js - you just have to add the "low tag" logic (output the IP with the lowest tag value, and then do a 'receive' from the input array port element that that IP came from).

Historically, Collate also adds open and close brackets to the output stream - this function could be packaged as a separate component, but it uses the same control fields as Collate, so it seemed practical to combine the two functions.

[tlrobinson]

In my quest to understand collate I implemented a version without brackets here: https://github.com/jpaulm/jsfbp/pull/17

I'll work on proof-of-concept with generators/promises/streams next.

[tlrobinson]

@jpaulm @alfa256 Here's a proof-of-concept using Streams, Promises, and ES6 Generators via Blubird's coroutine method: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with JavaScript and "promises" or "streams", but the relevant thing to note is I've implemented collate without using Fibers. Note that collate1 uses Generator functions, which will be available in ECMAScript 6, and I believe can be supported in older versions using a "transpiler". collate2 only uses Promises and should run in all browsers today, without any transpiling.

collate1 is a line-by-line port of "collate" that I implemented in JSFBP (https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the only changes are the addition of the yield statement in front of asynchronous operations (i.e. receive) and the "function " syntax to make it a ["generator function"](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function)

collate2 is admittedly less clear because it's written in an asynchronous style using Promises directly, but it proves "collate" can be implemented in JavaScript without fibers or even generators: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" (http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When in flowing mode, data is read from the underlying system and provided to your program as fast as possible. In paused mode, you must explicitly call stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive method to readable streams which returns a promise. You can use the promise directly (as I do in collate2) or yield it within a Bluebird coroutine if you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can implement a component in whichever style makes the most sense. From the outside it doesn't matter. For example, copier could be implemented as InputPort.openInputPort('IN').pipe(InputPort.openOutputPort('OUT'));

Streams also support "backpressure" so they can communicate to their upstream processes when they do or don't want more data: http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode" http://nodejs.org/api/stream.html#stream_object_mode but normal mode might be interesting in some cases.

[alfa256]

So from what I can see the api changes would be adding a yield to receive()s ? If that's the case then It'd be great, also I wonder if the yield could be even wrapped in the receive() ? In any case great job, this could be great for the project.

2015-02-17 16:47 GMT-03:00 Tom Robinson notifications@github.com:

@jpaulm https://github.com/jpaulm @alfa256 https://github.com/alfa256 Here's a proof-of-concept using Streams, Promises, and ES6 Generators via Blubird's coroutine method: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with JavaScript and "promises" or "streams", but the relevant thing to note is I've implemented collate without using Fibers. Note that collate1 uses Generator functions, which will be available in ECMAScript 6, and I believe can be supported in older versions using a "transpiler". collate2 only uses Promises and should run in all browsers today, without any transpiling.

collate1 https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L62-L101 is a line-by-line port of "collate" that I implemented in JSFBP ( https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the only changes are the addition of the yield statement in front of asynchronous operations (i.e. receive) and the "function " syntax to make it a "generator function" <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function>

collate2 https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L104-L151 is admittedly less clear because it's written in an asynchronous style using Promises directly, but it proves "collate" can be implemented in JavaScript without fibers or even generators: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" ( http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When in flowing mode, data is read from the underlying system and provided to your program as fast as possible. In paused mode, you must explicitly call stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L32-L48 method to readable streams which returns a promise. You can use the promise directly (as I do in collate2) or yield it within a Bluebird coroutine if you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can implement a component in whichever style makes the most sense. From the outside it doesn't matter. For example, copier could be implemented as InputPort.openInputPort('IN').pipe(InputPort.openInputPort('OUT'));

Streams also support "backpressure" so they can communicate to their upstream processes when they do or don't want more data: http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode" http://nodejs.org/api/stream.html#stream_object_mode but normal mode might be interesting in some cases.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-74738438.

[tlrobinson]

@alfa256 I don't believe yield can be moved inside of receive because it's a special keyword for use in generator functions (function* whatever(){}), but I don't think that's desirable anyway. Ideally the FBP kernel/framework should work without generator functions, but would support them as an extra layer of syntactic sugar in components.

Note that the only code in my proof of concept that's aware of generator functions or yield is the component itself. I think it should stay that way, but it's up for debate.

[brodavi]

Hi! Interested bystander here. I just finished reading the fbp book, and this jsfbp project excites me.

For what it's worth, I would like to second the motion for using streams. Streams have a large "following" (node, gulp, etc) and are relatively well understood even by pedestrian js developers like myself.

I share @tlrobinson's concern that node-fibers are off-putting. Streams, on the other hand, I can rally behind. Plus they are available in the browser today with a choice of packages, and hopefully native support soon.

Of course, there may exist very good reasons against using streams here, but my current (admittedly limited) understanding does not see any.

https://highlandjs.org/ https://github.com/substack/stream-handbook https://streams.spec.whatwg.org

[ComFreek]

The proof of concepts look _promis_ing! Note that Node.js 12.0 doesn't support generator functions (unless compiled with --harmony), so I switched to io.js in order to run your sample collate.js. It is essentially a Node.js fork with faster release cycles.

I have finished my changes which separate all Network-related thing from those related to (Fiber-)Runtime. I would be interested in whether streams or promises can be easily integrated, given the new separated Network/Runtime architecture. A Runtime (stored in core/runtimes/[name]/index.js) is a class which must implement these functions:

• pushToQueue(process): Pushes a process to the Queue
• run(processes, options, callback)
• setCallbackPending(callback)
• getCurrentProc()

Admittedly, the last two methods seem to still heavily rely on Fibers. I will look into their use cases and investigate.

I have read that Promise implementations might tend to create memory leaks (just search for "Promises memory leak" or "Promises garbage collection"). This is especially related to unresolved promises and/or errors. There is a very long discussion titled Promises/A+ implementations can lead to resource leaks in Node.

[tlrobinson]

I've expanded my proof-of-concept into a fully functional prototype: https://github.com/tlrobinson/sbp

Like the PoC, it uses Streams, as well as a combination of Promises, Generator functions, and Bluebird's coroutine (or Q's async) method to achieve a synchronous style similar to existing cFBP (classical FBP) implementations, including jsfbp.

I'd be curious to hear from the experts whether they'd consider this true "classical-FBP", and if not, why not?

[tlrobinson]

@jpaulm @alfa256 @ComFreek @brodavi @kenhkan Any thoughts on this?

[jpaulm]

Collate1 looks interesting - it seems strange that you can't "hide" the yield, but I guess we can just hold our noses!

What I am curious about is whether you ran a complete test case, e.g. fbptest01 with only the two changes you mention wrt collate1, and if so how did it perform relative to the version on JSFBP ( https://github.com/jpaulm/jsfbp/ ).

Collate2 looks much more alien to me - it even seems to have a recursive loop in it - or did I miscount the brackets?!

@brodavi You said, " I share @tlrobinson's concern that node-fibers are off-putting" . From a component or network builder's point of view, the only sign of fibers is one "require" - I don't think that should be "off-putting". The lack of support for fibers in browsers is apparently much more of a concern, but this will probably be remedied before long if and when fibers catches on!

Regards,

Paul M.

@jpaulm https://github.com/jpaulm @alfa256 https://github.com/alfa256 Here's a proof-of-concept using Streams, Promises, and ES6 Generators via Blubird's coroutine method: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with JavaScript and "promises" or "streams", but the relevant thing to note is I've implemented collate without using Fibers. Note that collate1 uses Generator functions, which will be available in ECMAScript 6, and I believe can be supported in older versions using a "transpiler". collate2 only uses Promises and should run in all browsers today, without any transpiling.

collate1 https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L62-L101 is a line-by-line port of "collate" that I implemented in JSFBP ( https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the only changes are the addition of the yield statement in front of asynchronous operations (i.e. receive) and the "function " syntax to make it a "generator function" <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function>

collate2 https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L104-L151 is admittedly less clear because it's written in an asynchronous style using Promises directly, but it proves "collate" can be implemented in JavaScript without fibers or even generators: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" ( http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When in flowing mode, data is read from the underlying system and provided to your program as fast as possible. In paused mode, you must explicitly call stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L32-L48 method to readable streams which returns a promise. You can use the promise directly (as I do in collate2) or yield it within a Bluebird coroutine if you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can implement a component in whichever style makes the most sense. From the outside it doesn't matter. For example, copier could be implemented as InputPort.openInputPort('IN').pipe(InputPort.openInputPort('OUT'));

Streams also support "backpressure" so they can communicate to their upstream processes when they do or don't want more data: http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode" http://nodejs.org/api/stream.html#stream_object_mode but normal mode might be interesting in some cases.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-74738438.

[tlrobinson]

The lack of support for fibers in browsers is apparently much more of a concern, but this will probably be remedied before long if and when fibers catches on!

@jpaulm It's highly unlikely fibers will be added to browsers. If you don't believe me try asking a few other JavaScript experts.

[jpaulm]

@tlrobinson And Streams and Promises are better than Fibers... how?! Because some JavaScript guru has blessed them...? Just curious!

[tlrobinson]

@jpaulm The main thing is with fibers you can't tell if a function call is going to block execution, whereas with generators you explicitly yield within a generator function (that's why the yield can't be hidden) Here's a description of the difference between generators and fibers: https://gist.github.com/creationix/6416266

But the important thing is generators are already in ES6 (the next version of JavaScript), and fibers would be mostly redundant, so they're unlikely to ever make it into the JavaScript spec and therefore browsers.

[jpaulm]

I just wrote the following to Comfreek:

send suspends when the connection is full, and receive suspends when the connection is empty. I don't see this as a leaky abstraction - the application designer does not even think in these terms - she just sees processes running asynchronously, with data chunks flowing across unidirectional connections - with back pressure.

That's why I keep saying it's a paradigm shift!

I don't want to know whether a function call is going to block execution

• it should not affect the logic! Go back and read Dijkstra on sequential processes! It's not rocket science! On Mar 7, 2015 7:32 PM, "Tom Robinson" notifications@github.com wrote:

@jpaulm https://github.com/jpaulm The main thing is with fibers you can't tell if a function call is going to block execution, whereas with generators you explicitly yield within a generator function (that's why the yield can't be hidden) Here's a description of the difference between generators and fibers: https://gist.github.com/creationix/6416266

But the important thing is generators are already in ES6 (the next version of JavaScript), and fibers would be mostly redundant, so they're unlikely to ever make it into the JavaScript spec and therefore browsers.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-77716351.

[tlrobinson]

In JavaScript it has always been safe to assume a function will run to completion before another event fires which could change the state that the function is using. Fibers break that assumption, which is unacceptable to a lot of people, including the people in charge of defining future versions of JavaScript.

Generators also break that assumption, but you have to explicitly opt-in by declaring a generator function (with the "function*" syntax) and use the "yield" keyword, so there's no chance of someone else's code deep in some library you're using yielding execution without your knowledge.

BTW this article expands on the gist I linked to in my last comment.

We can argue about which way is better, but it's not going to change the fact that you're unlikely to ever get fibers in the browser.

[jpaulm]

This discussion is interesting, but I feel it should be on the Group, so it gets a wider audience. Could one of you, @tlrobinson or @comfreek start a topic on the Group, maybe with the last few posts. ..?

On Mar 8, 2015 6:01 AM, "Tom Robinson" notifications@github.com wrote:

In JavaScript it has always been safe to assume a function will run to completion before another event fires which could change the state that the function is using. Fibers break that assumption, which is unacceptable to a lot of people, including the people in charge of defining future versions of JavaScript.

"Change the state that the function is using."

That's really the whole point of FBP : two functions cannot use the same "state ". It just seems odd to raise that objection in the context of FBP! Maybe an education problem?!

[jpaulm]

It seems to me that the appropriate place for much of this discussion is the JSFBP topic on the Group. Could we move at least the important posts there? Thanks.

PS I would do it, but it would be very difficult using my little tablet! :-) On Mar 8, 2015 8:21 AM, "Paul Morrison" jpaulmorr@gmail.com wrote:

This discussion is interesting, but I feel it should be on the Group, so it gets a wider audience. Could one of you, @tlrobinson or @comfreek start a topic on the Group, maybe with the last few posts. ..?

On Mar 8, 2015 6:01 AM, "Tom Robinson" notifications@github.com wrote:

In JavaScript it has always been safe to assume a function will run to completion before another event fires which could change the state that the function is using. Fibers break that assumption, which is unacceptable to a lot of people, including the people in charge of defining future versions of JavaScript.

"Change the state that the function is using."

That's really the whole point of FBP : two functions cannot use the same "state ". It just seems odd to raise that objection in the context of FBP! Maybe an education problem?!

[jpaulm]

Hi ComFreek,

Nice to hear from you - and nice to be back!

I am starting to look at JSFBP's derivatives - your streams-impl, and Tom's sbp (streampunk?). BTW Will you be updating your fork's readme to include the instructions you sent me? I think it would be useful.

What I find so strange is that these (to my way of thinking) contorted forms are said to seem more natural to a JS user (at least you and Tom seem to be saying this) than the nice simple syntax of the current JSFBP version (much improved by you, I admit!). In the former case, the user has to learn and get comfortable with Promises, Streams, etc., plus FBP, while in the case of basic JSFBP they just have to learn FBP!

I also don't see why 'yield' and those asterisks have to be visible - surely making them visible means that it could be meaningful to drop them - what happens if one or both of these are dropped by mistake?

Basically, it seems to come down to the fact that you and Tom don't believe that the JS community will accept node-fibers as readily as they have Promises etc. Surely that's just a matter of building a constituency of people who are more interested in getting the job done than in playing with abstruse concepts which probably don't have a well-defined use case anyway! Marcel @laverdet, are you getting any feedback on this?

ComFreek and Tom, I think I know where you stand on the issue of node-fibers vs. Promises, etc. - Ken, Alex, Ged, @bjouhier, @jeanhuguesrobert, @laverdet, I would be interested in hearing your views. I am double posting this note here, as I don't have email addresses for some of the people I would appreciate feedback from.

FYI The repositories I am looking at are:

https://github.com/jpaulm/jsfbp

https://github.com/ComFreek/jsfbp/tree/streams-impl

https://github.com/tlrobinson/streampunk.js

Best regards to all,

Paul M.

[tlrobinson]

Correct, my position is that it is very unlikely that fibers have a future in the JavaScript language standard (ECMAScript), and they can't even be reasonably shimmed into existing versions of JavaScript via "transpiling" (unlike generators or async/await). Therefore fibers will never work in browser JavaScript (the largest deployed programming environment, by far), and should not be used as the foundation of a general purpose programming framework if there is a suitable alternative. On Sun, Mar 29, 2015 at 12:01 PM jpaulm notifications@github.com wrote:

Hi ComFreek,

Nice to hear from you - and nice to be back!

I am starting to look at JSFBP's derivatives - your streams-impl, and Tom's sbp (streampunk?). BTW Will you be updating your fork's readme to include the instructions you sent me? I think it would be useful.

What I find so strange is that these (to my way of thinking) contorted forms are said to seem more natural to a JS user (at least you and Tom seem to be saying this) than the nice simple syntax of the current JSFBP version (much improved by you, I admit!). In the former case, the user has to learn and get comfortable with Promises, Streams, etc., plus FBP, while in the case of basic JSFBP they just have to learn FBP!

I also don't see why 'yield' and those asterisks have to be visible - surely making them visible means that it could be meaningful to drop them - what happens if one or both of these are dropped by mistake?

Basically, it seems to come down to the fact that you and Tom don't believe that the JS community will accept node-fibers as readily as they have Promises etc. Surely that's just a matter of building a constituency of people who are more interested in getting the job done than in playing with abstruse concepts which probably don't have a well-defined use case anyway! Marcel @laverdet https://github.com/laverdet, are you getting any feedback on this?

ComFreek and Tom, I think I know where you stand on the issue of node-fibers vs. Promises, etc. - Ken, Alex, Ged, @bjouhier https://github.com/bjouhier, @jeanhuguesrobert https://github.com/jeanhuguesrobert, @laverdet https://github.com/laverdet, I would be interested in hearing your views. I am double posting this note here, as I don't have email addresses for some of the people I would appreciate feedback from.

FYI The repositories I am looking at are:

https://github.com/jpaulm/jsfbp

https://github.com/ComFreek/jsfbp/tree/streams-impl

https://github.com/tlrobinson/streampunk.js

Best regards to all,

Paul M.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-87455434.

[JeanHuguesRobert]

Hi Paul,

The lack of threads, actually the lack of "continuations", aka callcc(), is definitely an issue that puts JavaScript in the unfortunate situation where C was before threads where introduced, a while ago, 25 years ago I guess.

I tend to think that the current situation will prevail for the next 5 years at least, forever maybe. The reasons for that are beyond my understanding (and too long to explain) but they are strong.

This is Gravity.

Consequently the solution is not to rely on threads.

Which means either A/ async everywhere or B/ transpiling somehow.

For obvious performance reasons, A is probably the solution for anything "core". For less critical code or "user land", some B might work but I would rather not be in position to select which B solution to use, better let that decision to the application developer.

If I were you I would build low level efficient abstractions to get my job done and I would provide "bridges" to interface with callback(err,..)/Promise/Generator/Worker/ServiceWorker/WhatEverNext application code.

[bjouhier]

@jpaulm Hi Paul,

I've been monitoring the fibers/promises/generators/async-await debates for the last 5 years and I concur with the general opinion that fibers, despite all its qualities, is very unlikely to get a very wide acceptance.

Fibers had a window of opportunity in 2011-1013 and it managed to gather some interesting momentum, especially with meteor. There were lots of passionate debates about it at the time, and many node.js top guns advocated strongly against it. At the time, fibers did not have many contenders but the landscape is changing: generators and promises are now baked into V8 (ES6) and async/await is lined up for ES7. Once JS gets promises + async/await, the main pain points of async programming (callback hell) will be addressed by vanilla JS and fibers will be harder to sell.

From a language design standpoint, there is a fundamental difference between fibers and the mechanisms that are being added to ES: fibers provide true (deep) continuations while generators and async/await only provide shallow (single frame) continuations. The shallow model forces all the yielding points to be explicit, which is an advantage when reasoning about concurrency issues (you don't have to dive into sub-functions to find out where a function may yield). The designers of the language are obviously strongly attached to this model and I don't foresee this changing any time soon.

[forresto]

Here is an example of 4800 'green' threads running in the browser via clojurescript and core.async. The Core.Async talk linked there is quite interesting: http://programming-enchiladas.destructuring-bind.org/rm-hull/8262502

[jpaulm]

Thanks, Forrest, maybe Clojure is the way to go! Interesting that he talks about addressing "callback hell" as well - this seems to be the most common criticism of JavaScript!

On Fri, Apr 3, 2015 at 11:08 AM, Forrest Oliphant notifications@github.com wrote:

Here is an example of 4800 'green' threads running in the browser via clojurescript and core.async. The Core.Async talk linked there is quite interesting: http://programming-enchiladas.destructuring-bind.org/rm-hull/8262502

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-89314837.

[jpaulm]

Merci, Jean-Hugues et Bruno!

I assume all these mechanisms (Promises, Streams, etc.) are essentially green-thread-based. Given that more and more devices have multiple cores, perhaps green thread solutions are no longer as relevant as they were...? What approaches are there to having JavaScript play with multiple cores?

Also, Bruno, you say, "The shallow model forces all the yielding points to be explicit, which is an advantage when reasoning about concurrency issues (you don't have to dive into sub-functions to find out where a function may yield). " From the point of view of FBP, that's exactly what you don't want! A function can only yield on a suspended send or a suspended receive. Period. All data is either in an IP or in a function's local (stack) storage. So no fancy reasoning involved!

I'm curious how Promises work with stacks - I assume a Promise uses the next available space in the stack but has access to data above it in the stack. It sounds like Clojure has figured out how to manage Promises with multiple stacks...

[bjouhier]

Promises and streams don't use green threads. They are usually (*) implemented as vanilla JS libraries. Promise libraries can be a bit hairy because the Promise specs impose a rich API but you can capture the essence of a promise (a future aka thunk) in a few lines of JS. I blogged about it a while ago (https://bjouhier.wordpress.com/2011/04/04/currying-the-callback-or-the-essence-of-futures/).

(*) promises are now part of the language itself (ES6) so they can be implemented in C and provided directly by the JS engine. But this does not change the threading model.

With promises, all JS code executes on the main stack. The promise (or future) encapsulates a callback and memorizes the result or error when the callback completes.

Streams are similar; they execute on the main stack and they encapsulate various events (node.js style streams) or a simple callback (my ez-streams library).

Generators go one step further. They still execute on the main stack but they capture one stack frame when the generator is suspended (with yield). That stack frame is pushed again on the stack when the generator is resumed (with run). So there is still only one thread, no green threads. The single frame continuation rule imposes that you yield only at top level in your generator function; you cannot yield from a subfunction. This way there is only one frame to save to capture the state of the generator.

Fibers are different. Each fiber has its own stack and yield may be called at any depth in a fiber's stack. Fibers are green threads.

Node uses more than one thread but all JS code is executed by a single thread with an event loop. C++ add-ons may use additional threads to execute operations while the JS thread is running but they always report their completion by posting a callback to the main loop which is run by the main JS thread.

There had been attempts to provide true (preemptive) JS threading in node.js. Threads-a-gogo was an early attempt. This has been revived recently by workers in io.js: https://github.com/iojs/io.js/pull/1159.

The model behind workers is different from the model behind threads in Java or .NET: each worker runs in its own isolate. You can pass messages between the main JS thread and worker threads but you cannot share mutable state between threads. Worker threads don't block; they perform their I/O asynchnously and they each have their own event loop to handle their I/O callbacks. This share nothing model is, IMO, much easier to work with than classical threading with shared state and monitors.

[bjouhier]

One more word on the shallow model and FBP: of course you would be on the safe side with FBP because the yielding points are imposed and hidden. You are also on the safe side if you use Marcel's futures library with the right discipline (the yielding point are wait() calls that surface at all levels). But the JS language designers have to cater for every situation, not just FBP or a specific futures library. This is why they are so attached to the single frame continuation rule.

Maybe you should take a look at Go. Green threads may be pariahs in JS but they are first-class citizens in Go; they are called goroutines.

[jpaulm]

Hi @bjouhier, thanks for the info on Promises - that helps quite a bit! I can of course see the single frame continuation *rule - *in fact, I have long suspected that one could build a simpler FBP engine by restricting sends and receives to the top level of a process. I might play with that! I am not really keen, though, on leaking implementation restrictions into the mental model - but I can see that it might be expedient!

You might be interested in looking at Vladimir Sibirov's Goflow - https://github.com/trustmaster/goflow .

You mentioned Threads à gogo - doesn't that also require a native module - which I thought was one of the problems with JSFBP being dependent on Marcel's node-fibers...? If you don't do something like that, how do you run JavaScript on a multiple-core machine? I suppose you could run JavaScript on one or more Boost threads...?

Regards,

Paul

On Sat, Apr 4, 2015 at 4:59 PM, Bruno Jouhier notifications@github.com wrote:

One more word on the shallow model and FBP: of course you would be on the safe side with FBP because the yielding points are imposed and hidden. You are also on the safe side if you use Marcel's futures library with the right discipline (the yielding point are wait() calls that surface at all levels). But the JS language designers have to cater for every situation, not just FBP or a specific futures library. This is why they are so attached to the single frame continuation rule.

Maybe you should take a look at Go. Green threads may be pariahs in JS but they are first-class citizens in Go; they are called goroutines.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-89666204.

[tlrobinson]

Threads-a-go-go (and specifically this fork: https://github.com/audreyt/node-webworker-threads ) provides an API similar to "WebWorkers" which are in most browsers.

Unfortunately node-webworker-threads seemed very unstable last time I tried it (a few weeks ago)

As an aside, I've got a toy FBP implementation in ~100 lines of Go here: https://github.com/tlrobinson/streampunk.go On Sat, Apr 4, 2015 at 4:18 PM jpaulm notifications@github.com wrote:

Hi @bjouhier, thanks for the info on Promises - that helps quite a bit! I can of course see the single frame continuation *rule - *in fact, I have long suspected that one could build a simpler FBP engine by restricting sends and receives to the top level of a process. I might play with that! I am not really keen, though, on leaking implementation restrictions into the mental model - but I can see that it might be expedient!

You might be interested in looking at Vladimir Sibirov's Goflow - https://github.com/trustmaster/goflow .

You mentioned Threads à gogo - doesn't that also require a native module - which I thought was one of the problems with JSFBP being dependent on Marcel's node-fibers...? If you don't do something like that, how* do* you run JavaScript on a multiple-core machine? I suppose you could run JavaScript on one or more Boost threads...?

Regards,

Paul

On Sat, Apr 4, 2015 at 4:59 PM, Bruno Jouhier notifications@github.com wrote:

One more word on the shallow model and FBP: of course you would be on the safe side with FBP because the yielding points are imposed and hidden. You are also on the safe side if you use Marcel's futures library with the right discipline (the yielding point are wait() calls that surface at all levels). But the JS language designers have to cater for every situation, not just FBP or a specific futures library. This is why they are so attached to the single frame continuation rule.

Maybe you should take a look at Go. Green threads may be pariahs in JS but they are first-class citizens in Go; they are called goroutines.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-89666204.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-89682442.

[jpaulm]

Node-webworker-threads looks interesting! However I think I'm missing something important - https://github.com/audreyt/node-webworker-threads/blob/master/README.md - Why Threads (4) says "The threads of a process share exactly the same address space, that of the process they belong to. Every thread can access every memory address within the process' address space.", but @bjouhier above said "You can pass messages between the main JS thread and worker threads but you cannot share mutable state between threads." Are we talking about different entities? Or is immutability enforced somehow? Probably a dumb question...?! :-)

[bjouhier]

@jpaulm This sentence is a generic statement about threads. I doubt that it applies to JS code run by node-webworker-threads. With V8 if you try to have two threads that run in the same isolate and that both modify a shared JS object, you will just get a crash. The V8 engine is not designed for that.

[jpaulm]

How would the machine know to crash?! Von Neumann machines aren't usually smart enough...

@audreyt would you care to comment? I read the passage about threads in https://github.com/audreyt/node-webworker-threads/blob/master/README.md as being very specifically about worker threads...? TIA

[bjouhier]

What I meant is that you will get an assertion error or a SEGV or another kind of error and your process will terminate. Simply because you are calling an API from more than one thread and that API is not designed for that.

[jpaulm]

Ah, I see! Thanks, @bjouhier! Back to square one!

On Mon, Apr 6, 2015 at 2:33 AM, Bruno Jouhier notifications@github.com wrote:

What I meant is that you will get an assertion error or a SEGV or another kind of error and your process will terminate. Simply because you are calling an API from more than one thread and that API is not designed for that.

— Reply to this email directly or view it on GitHub https://github.com/jpaulm/jsfbp/issues/14#issuecomment-89945006.

[ComFreek]

@jpaulm The more I re-read some of the comments above, the more I share the conclusion that a FBP implementation allowing its graph to span across different networks, operating systems and programming languages would definitely be superior to any FBP implementation restricted to a single programming language. As you can see, JSFBP will not be able to use real threads in near future as JS (and V8 as the runtime engine for Node.js) are simply not designed for that use case.

So, I'd like to throw in an idea, which has probably already been discussed in the FBP group or elsewhere:

Create a FBP library providing a client and server interface for receiving and sending packages, respectively, to the outer world. The connection shall be treated as general as possible, thereby allowing a wide range of connection types, starting with simple memory buffers storing the IPs and ranging up to mobile data connections as used in current smart phones.

[jonnor]

@ComFreek https://github.com/the-grid/msgflo

[chadrik]

I've also wondered if it would be possible to do this with a FBP wrapper around celery and its canvas primitives.

[audreyt]

node-webworker-threads works on Node 0.10.x and is known to be unstable with 0.12.x and io.js, which I hope to fix in the coming weeks.

It is true that node-webworker-threads cannot share mutable structure across worker threads and with the main thread; the same principle applies to WebWorker in browsers, too.

[jpaulm]

@comfreek As always, my questions would be: what language(s) and what operating system(s)? Given the existence of languages which don't play together too well, such as Java and C#, sockets would seem a promising approach... JavaScript seems to be able to talk to C[++], so this suggests that it might be interesting to work with Boost... What did you have in mind?

[laverdet]

fwiw I want to echo what @tlrobinson & @bjouhier said about fibers never making it into browsers or a standard. I started working on fibers in early 2011 when early versions of generators were still being discussed for standardization. We all knew yield was coming to JS and that it would likely be single frame, but we didn't know the exact API or syntax. And it definitely wouldn't be ready soon. I wanted yield right then and there and full continuations were the easiest way to build it.

[ComFreek]

@jonnor Thanks for the link, the readme seems very elaborate! Is there any quickstart guide available? I would like to see it in action (e.g. having component A running on my PC and component B on my notebook, both connected to my router).

@jpaulm As it is the very idea of the "FBP library", I would target some a wide range of operating systems and programming languages by employing low-level languages, such as C and C++, and common network protocols.

I'll have a look at celery, thanks for the link @chadrik.

[jonnor]

@ComFreek we're using it production a couple of places, but still have not landed on final API/protocol - so usage docs are not the priority just yet. Will probably tag & announce first stable in a couple of weeks, when we've migrated everything in the production pipeline to it. For now, see: https://github.com/the-grid/msgflo/blob/master/spec/fixtures/participants.coffee https://github.com/jonnor/imgflo-server/blob/master/src/worker.coffee