Branching logic?

[tmenier]

I'm test driving this library to implement a data processing pipeline. A few steps into the pipeline definition I need to branch, i.e. "if x then write to channel A else write to channel B". (Specifically, it's a validation step, with channel B appending failures to a log currently with successes moving on to the next step.) Am I correct that there isn't any direct support for this in the library? Is it something you might consider as a new feature? I'm envisioning something that takes a predicate, similar to Filter except the failed results are somehow sent in a different direction rather than discarded. (Fork perhaps?) Or maybe you can suggest a less direct approach that doesn't diverge too far from the patterns established by the library?

Thanks in advance!

[electricessence]

Interesting you bring this up. We just had an iteration for something like what you're mentioning. First off, .Filter uses a reader to "pull" items out of the channel in similar way to how .Where works with LINQ. .Filter doesn't handle exceptions and so you have to be careful as you could break your pipeline.

That said, you might want to read this conversation (and it's result): https://github.com/Open-NET-Libraries/Open.ChannelExtensions/pull/35

And here's a merge you might be interested in: https://github.com/Open-NET-Libraries/Open.ChannelExtensions/commit/e54bc53a1f3854ae11da9a6e075ed02fb089a3e3

All that said, you might find System.Threading.DataFlow a better option for more complex scenarios and what you're trying to do.

[tmenier]

Cool, I"ll check these out, thanks for the quick reply!

[tmenier]

Looks like the conversation was very relevant and PipeFilter is pretty close to what I was looking for. I absolutely shared your hesitation for introducing the out param, as it sort of makes the "else" condition a second-class citizen, which may not be the case at all. But I also see how it's hard to come up with a "fluent" alternative that doesn't descend into nesting hell. FWIW (and I'm sure it's too late), this is a very naive initial idea that was rolling around in my head:

.Split(x => x > 0)
.Matched(channel => channel
  .Pipe(...)
  ...)
.Unmatched(channel => channel
  .Pipe(...)
  ...)

Obviously doesn't take into account details like channel options and async variations, just sort of an imagining of an alternative API.

[electricessence]

Applause! You get it. It wasn't an easy choice. But it seemed like an ok way to go. At some point, I say to people, if you want complex, just start using DataFlow. :P TBH, the current version of this library is at a very stable state and I prefer being very careful/reserved about updates/adds. Ideas need to be well thought out, and somewhat in demand.

So does PipeFilter suit your needs, or no?

[tmenier]

So does PipeFilter suit your needs, or no?

Probably. I'll likely either use that or write to the 2 channels directly, not sure yet which will be cleaner in my case. I may not get back to it until mid-week but I'll report back when I do. Thanks again!