Higher-order components as STDlib API Pattern (FP in FBP)

[emil14]

Neva supports interfaces and (static) dependency injection, we investigate how handy it is to use that pattern to handle cases like map/filter/reduce or even scenarios like working with resources that needs to be closed (kinda like passing callbacks in FP languages)

Example

Let's imagine ForEach component that accepts IHandler interface as a dependency. It's goal to do something for every element of a stream (list, map, etc).

interface Handler {
    <T>(v T) (v T)
}

component ForEach<T>(seq maybe<T>) (seq maybe<Y>) {
    nodes { Handler<T>, Unwrap<T>, Wrap<Y> }
    net {
        :seq -> unwrap
        unwrap:some -> handler -> wrap -> :seq
        unwrap:none -> :seq // FIXME order is not guaranteed
    }
}

component PrintEachInt(seq maybe<int>) (seq maybe<int>) {
    nodes {
        printEvery ForEach<int> { Println<int> }
    }
    net { :seq -> printEvery -> :stop }
}

P.S - please note that this implementation is naive (I'm talking about the FIXME part), but that's another thing to discuss.

[emil14]

Related to #575

[emil14]

This how type-safe e.g. Map could look like

interface IMapper<T, Y>(data T) (res Y)

component Map<T, Y>(data stream<T>) (res stream<Y>) {
    nodes { mapper IMapper<T, Y>, builder Struct<StreamItem> }
    net {
        :data.data -> mapper -> builder:data
        :data.idx -> builder:idx
        :data.last -> builder:last
        builder -> :res
    }
}

component Main(start) (stop) {
    nodes {
        Map<int, string> { IntToStr<int, string> }
    }
    net {
        // ...
    }
}

component IntToStr<T int, Y int>(data T) (res Y) {
    // ...
}

Problem

To provide type-safety (because of how Nevalang's type-system works) we have to

component IntToStr<T int, Y int>(data T) (res Y)
...
Map<int, string> { IntToStr<int, string> }

Instead of just

component IntToStr(data int) (res string)
...
Map<int, string> { IntToStr }

Weak solution

It's possible to replace type-parameters in the interface with any. This makes everything simpler:

interface IMapper(data any) (res any)
...
nodes { mapper IMapper }
...
component IntToStr(data int) (res string)
...
Map<int, string> { IntToStr }

But the problem is that we can now pass anything as a Map's dependency (as long as it has data and res ports), types could be anything, we won't get compiler error if they are not compatible with what we've passed to Map<T,Y> type parameters.

[emil14]

Strong solution

We need somehow tweak analyzer/type-system so

IHandler<T, Y> (data T) (res Y)

Is considered compatible to

SomeComponent(data int) (res string)

[emil14]

Added critical because it can influence language design

[emil14]

These APIs are BLOCKING

As soon as I got this shit working I understood the scariest possible thing in the Universe

Shouldn't unblocking API be first-class citizen? I mean we could implement map/filter/reduce/for/you-name-it but they block. Like, if you pass a stream to for, then flow gonna be blocked until all the elements are processed.

Ofc we could implement this and sell it like a "easy default way of solving problems" and if you'll need extra perf then use some more "advanced" API (e.g. from streams package). That makes sense and sounds ok but... Hey? Should't we make the default way of solving problems performant?

I know perf is not goal at Alpha but it's about streaming, not some CPU cycles... Okay enough words. Lemme show some code

Blocking API example

Let's say we wanna map each element of a list and then perform some side-effect (we gonna print for simplicity)

component Main(start) (stop) {
    nodes { lists.Map{Heavy}, PrintEach }
    :start -> ($lst -> map -> printEach -> :stop)
}

Everything is ok but the program could be faster if map wasn't blocking. We gonna first map the whole list (let's say it's 100 of deeply nested objects, let's say that would take us 100s), then we gonna print each object. So _first mapped object gonna be printed after 100s.

Non-blocking API example

The idea is to have wrapper, that does safe unwrapping of a stream-item inside, just like previous version, but instead of blocking until all stream processed it wraps

component Main() () {
    nodes { lists.Map{Heavy}, For{Println} }
    $l -> map -> for.last -> if:then -> :stop
}

• lists.Map takes list, iterates over it, on each iteration "calls" mapper (Heavy in this case), then "wraps" result into stream item and sends downstream
• For receives a stream-item, calls handler (Println) with it and sends downstream
• We select .last field of each item and send to if node, if last==true it sends to ->:stop and terminate, otherwise nothing happens (we don't use if:else so these messages just deleted)

First mapped object gonna be printed after 1s which is x100 faster

[emil14]

These APIs are BLOCKING

But maybe you can just pass handler inside For that will do the mapping AND side-effect in that case? Passing some custom component.

Also should be possible (even tho not idiomatic) to pass mapper with side-effect?

[emil14]

I'll close this one because a few APIs are already in main and it's clear where to go next

• FP + FBP = <3 (this is the way)
• By FP we mean higher order components that works with streams and/or collections
• No problems with blocking APIs, there are non-blocking alternatives that are not something synthetic but rather natural (Like Map)