Iterator protocol (early termination for stream processing for both parent and child)

[emil14]

Go Example

In Go we can break the loop to stop next scanner.Text() calls.

func main() {
    stream := strings.NewReader("line1\nline2\nline3\nstop\nline4\n")

    scanner := bufio.NewScanner(stream)

    for scanner.Scan() {
        line := scanner.Text() // this is our `->:next`
        fmt.Println("Received:", line)

        if line == "stop" { // this is our `->:done`
            fmt.Println("Terminating the loop.")
            break
        }
    }
}

Can we do something like that in Neva? It made me think about termination of a stream processing. It seems like if we have stream we need take it all. Yes we can ignore some elements e.g. process first 10 and ignore all the others but upstream will still produce those elements. This is the core idea of this - can we make upstream stop?

Idea

// Iterator allows you do early return while you process stream
// The difference with just ignoring the rest of the stream elements
// Is that you can actually stop the stream from happening
// It's important because it means that upstream component can actually stop producing new items
flow Iterator<T>(chunk T) (next any, done any) {
    // receiver chunk
    // send to next if you want next chunk
    // send to done if you done

    // after you send to next you'll receive either
    // next chunk or stream will be ended

    // if you'll send to done stream will be ended
    // if there was elements in the stream they will be omited
}

Ofc to make it work upstream component must support this "protocol". This is two-way communication and in dataflow we need a loop for that (unlike call-return model which is sync+bidirectional).