Suggestion : "Lazy iterators" like "()" in addition to "[]" as in Python

[serge-hulne]

Proposal

• Create a second syntax to handle data coming from streams (files, web) as lazy operators:

• () vs [] as in python

• Example:

A = [f(x) for x in V]

A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.

However if one uses:

A = (f(x) for x in V)

Background

In Python if one uses the structure:

A = [f(x) for x in V]

A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.

However if one uses:

A = (f(x) for x in V)

Then A is a mere iterator and the values will only be computed as needed.

• With the array notation "[]", in a cascade of list comprehensions, all the intermediary results will therefore be kept in RAM.

• If one works with a very large structure and does a large number of successive transformations on it (typically in the field of NLP (see FLTK, for instance)), then the set of intermediary results can cause excessive usage of memory.

• Exemple of code where a "lazy operator" (implemented as a channel) gets transformed into an array by the current goplus handling of list comprehension

import "fmt"

type Counter struct {
    Value int
    HasNext bool
}

func (c *Counter) Next() {
    if c.Value <10 {    
        c.Value++
        c.HasNext = true
    } else {
        c.HasNext = false
    }
}

func GeneratortoIterator(c Counter) chan int {
    ch := make(chan int)
    go func() {
        defer close(ch)     
        for c.HasNext {
            ch <- c.Value
            c.Next()
        }
    }()
    return ch
}

func main() {

    c := Counter{0, true} 

    it := GeneratortoIterator(c) 
    // it is a channel (lazy iteraor)

    v1 := [x for x <- it]
    // v1 is an array at this point

    fmt.Printf("Type: v1 %T\n", v1)

    for k, v <- v1 {
        println "v[", k, "] = ", v
    }

    v2 := [x * 2 for x <- v1]

    println "v2  = ", v2

}

Workarounds

One way to code it consists in substituting arrays with channels, as in my example, above.

[serge-hulne]

Here is one possible implementation in Go for lazy iterators and associated functions (filter, map, reduce, etc.):

https://github.com/serge-hulne/go_iter

PS: I refactored it to make it easier to use.