Feature request: Option to get synchronous execution of processes

[samuell]

Just a placeholder for the idea that @trustmaster came up with during the chat today, to add an option to make a process execute synchronously, thus following the FIFO pattern typical of many FBP systems.

[samuell]

I definitely subscribe to this idea. Being able to mix and match between synchronous and asynchronous processes, to use the best of both worlds, could be a killer feature of GoFlow, I imagine. Can't wait to try this out :)

[trustmaster]

This should be easy to implement and synchronous components won't need StateLock mutex.

[trustmaster]

So far I have implemented a per-component switch and a global switch for all components in the app. By default all processes are asynchronous.

Synchronous process creation example using Mode property of flow.Controller:

printer := new(Printer)
printer.Component.Mode = flow.ComponentModeSync

The default mode for all components can be switched before processes are created using a global variable:

flow.DefaultComponentMode = flow.ComponentModeSync
// All processes started from here will be synchronous

[samuell]

Cool, will test it out immediately!

[samuell]

Nice, now it works if using unbuffered channels! (I get back the exact same result)

When using buffered channels though, I get very strange ordering of the lines!

I tried to add numbers to the beginning of the lines, so that my input file now looks like this: https://gist.github.com/samuell/6202722

... and then, with buffer size of 512, I get output looks like this :-o : https://gist.github.com/samuell/6202726 (So, completely different order, with line 99 even occuring two times, one time in the beginning, and one in the end)

This is my current version of basecompl_blow.go: https://gist.github.com/samuell/6202714

One could of course just run with unbuffered channels, but it's a bit of a pity, since some performance results I did, tends to show that performance is much better when having a bit of buffer ...:

[samuell]

Hmm, with some logging in the basecomplementers (tagging with bc[n]), I see that the wrong order is gotten already in the first basecomplementer: https://gist.github.com/samuell/6202757 ... and that yes, the correct order is gotten in the file reader, so it is somewhere between the file reader and the first basecomplementer, that the shift in ordering happens.

I wonder if we're running into this:

"it is possible in the gc implementation for channel send or receive operations to jump the queue under various simultaneity conditions. For example, two sends waiting on a buffered channel that wake up in response to two receives might end up sending in a different order than they queued. But since the sends are waiting at the same time, neither can be considered to happen before the other, so you'd never be able to tell." (https://groups.google.com/forum/#!topic/golang-nuts/CPwv8WlqKag ) (last mail)

Situation improves a lot if I remove the buffering on between the file reader and the first basecomplementer. Then only one line is shifter out of order (as opposed to like over 50%, before)

Btw, I added some logging in each of the components, and pushed that to a new branch, "debug", of the blow library.

[trustmaster]

This doesn't sound good, however

The main way this could be observed is that if you have one sending goroutine sending a sequence of values to one receiving goroutine, the sequence will arrive in order.

And in Sync mode the situation should be exactly this: one goroutine sending, one goroutine listening. What I want to check next is how a plain Go program behaves.

[trustmaster]

I've implemented the basecomplementer program in plain Go: https://gist.github.com/trustmaster/6204162. It results into exactly the same mess as the GoFlow one. At that I've noticed that even with BUFSIZE = 1 some lines are messed.

[samuell]

@trustmaster Oh ... (thanks for testing ... I should of course have tested this myself ...)

Good to know the problem is not in GoFlow at least!

[trustmaster]

If you don't see anything suspicious in the code, then it's worth asking on the golang nuts mailing list. Such channel behavior is neither normal nor reliable.

[samuell]

@trustmaster Yea, will try to make a few more tests, just to be really sure of the minimal case that produces this, and then ask about it.

[samuell]

Posted a question now: https://groups.google.com/d/topic/golang-nuts/5elNYb5J-aI/discussion

[trustmaster]

@samuell Thanks for that topic!

BTW, have a look at the issue #8

I'll close this issue for now since this mission is complete :)