F# API?

[Szer]

Streams are very FP-friendly concept. But C# is not very FP friendly language, that's why making ETL processes with F# is much easier (currying, pipelining etc).

Do you have any plans on making F# friendly facade?

[cybertyche]

We have plans to do so, yes. We almost delayed going OSS until we had one, but decided in the end it was better to go public sourcing sooner rather than later.

Can you give an example of what such an ETL process would look like expressed in F#? I can then see how close we come to it out of the box over the winter break, and then maybe queue up a work item to flesh it out.

[Szer]

@cybertyche

sure!

here is example of my usual ETL with Hopac.Streams (syntax won't differ much with Nessos.Streams or AsyncSeq)

Explanation:

• dataLake.Download - returns Job<T> value. This is like Task<T> in C# but slightly different.
• >>- - monadic map. Everything here is in Job monad. Closest example .ContinueWith(x => ...) where lambda inside returns non-task value.
• >>= - monadic bind. This is like .ContinueWith(x => ...) where lambda inside returns task value. Similar to Haskell :)
• >> - function composition. No example in C#. it takes function a -> b and function b -> c to produce function a -> c
• |> - trivial pipeline operator. It's behaviour could be achieved with writing a lot of extensions methods in C# which are not required in F#. Currying power.
• ChStream.ofSeq - creates async stream from IEnumerable
• mapParallelJob - takes degree of parallelism and lambda (x -> Job<y>) to produce stream of y
• collectFun - takes stream of x and lambda (x -> seq<y>) to produce stream of y
• mapFun - takes stream of x and lambda (x -> y) to produce stream of y
• chooseFun -takes stream of x and lambda (x -> y option) to produce stream of y (filtering None values)
• filterFun - takes stream of x and predicate lambda (x -> bool) to produce stream of x (filtering falses)
• foldToList - takes stream of x and creating single immutable linked list of values. Because it always prepending new values to resulted list, it is memory efficient (no need to reallocate)
• run - blocking thread and waiting for result. This is single place of awaiting whole chain of monadic binds and maps.

Every Stream method has two version. Fun version. like mapFun, filterFun, collectFun

x -> y x -> bool x -> seq<y>

and Job version. like mapJob, filterJob, collectJob

x -> Job<y> x -> Job<bool> x -> Job<seq<y>>

this helps to easily build pipelines with a lot of async calls. Streams themselves helps to speedup IO processes and CPU bound operations because they downstream data immediately.

F# helps to build such pipelines (and I have a LOT of them) much faster with consice source code.

[cybertyche]

I had a conversation with a colleague, and I think this is something we will definitely do. We're going to ponder a path of execution over the course of the holiday break and likely have something in January.

[pshrosbree]

@Szer Are you thinking of something along the lines of a module that wraps the Trill API with convenience methods?

[Szer]

@pshrosbree exactly! Facade like Akkling for Akka.Net or Orleankka for Orleans would be enough!

Yes it could be community-driven nuget, but it's always better to get it out of the box with first-class support

[pshrosbree]

OK, so the PR is the sort of thing you're looking for? Obviously extensively fleshed out, but focusing on idiomatic F#?

As mentioned in the PR comment, it would be nice for Trill to leverage F# quotations in addition to Expressions, but that will require more work than a simple facade.

Akkling has a builder. It might be nice to come up with a good builder for Trill, perhaps something along the lines of the various Rx builders.

[Szer]

@pshrosbree do you want me to prepare such PR? I could, but I have to dive into Trill first to produce high-quality PR :)

[pshrosbree]

Sure. You could contribute to https://github.com/Microsoft/TrillSamples/pull/5 or start something yourself. We'd be interested in any contributions. You may want to start with some core operators, rather than attacking the entire API surface.

What would be interesting is to replace the use of Expression in Trill with F# quotations and extend this support deep into to Trill. This could be done either directly or by making quotations into Expressions. There is a library that maps quotations to Expressions but it is limited.