No easy way of predictable termination when working with streams (sequences)

[emil14]

Problem

Let's take a simplest problem - "print each number in a stream". Turns out current implementation doesn't allow one to do so without introducing some extra complexity.

Demonstration

Here's naive attempt to program this logic:

$lst -> iter -> unwrap
unwrap:some -> println -> del
unwrap:none -> :stop

The problem is with this line unwrap:none -> :stop.

There's no guarantee that unwrap:none is sending after all elements from unwrap:some are printed. In fact, there's higher chance that we terminate before we print all the elements because printing takes time.

The root of this problem is parallel processing without synchronization. If only we could somehow await for all elements to be printed.

Bad solution

One way to fix this is to get rid of parallelism and introduce a pipeline. If data moves through the single flow then there's no asynchronism and everything happens in sequential manner.

Let's print messages before unwrapping them!

$lst -> iter -> println -> unwrap
unwrap:none -> :stop

Because every message that comes to -> unwrap is already printed there's a guarantee that when unwrap:some -> is sending, all values are processed (printed in this case).

However, there's a problem with this solution. Let's say our $lst is [1,2,3]. The output will be not 1\n2\n3\n, it will be 1\n2\n3\n<nil>\n! It could be even worse if we change how maybe is represented as a string in the future. In other words - this is not a solution because program operates in a different way.

---

Important to understand that the problem is not about maybe, it's about parallelism and lack of synchronization. However, changing how stream works OR maybe how Iter works could fix this. As well as changing something else.

[emil14]

Use structs with indexes instead of maybe

Imagine each stream's element is wrapped into something like this

pub type StreamItem<T> struct {
  data T
  idx int
}

Component that emits stream uses something like this StreamItem | StreamDelimiter

StreamDelimiter could be {idx int} or even just int

Lock

When unwrap:none (or it's alternative) is sending we lock until some "processor" is done with the message from unwrap:some (or it's alternative). The trick is to unlock only if message from :some and :none has the same index (or if they are equal themselves, depending on the API)

[emil14]

^ question is how we handle maps this way?

(we can use indexes as well)

[emil14]

When this will be implemented/fixed it might be possible to #568

[emil14]

This is how stream item could look instead of maybe. If handlers will receive it instead of just data we can get rid of parallelism (but the API for user won't be as handy)

pub type stream<T> struct {
    idx int
    last bool
    data T
}

[emil14]

Closed in favor of #644