The Go+ programming language is designed for engineering, STEM education, and data science. Our vision is to enable everyone to become a builder of the digital world.
Create a second syntax to handle data coming from streams (files, web) as lazy operators:
() vs [] as in python
Example:
A = [f(x) for x in V]
A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.
However if one uses:
A = (f(x) for x in V)
Background
In Python if one uses the structure:
A = [f(x) for x in V]
A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.
However if one uses:
A = (f(x) for x in V)
Then A is a mere iterator and the values will only be computed as needed.
With the array notation "[]", in a cascade of list comprehensions, all the intermediary results will therefore be kept in RAM.
If one works with a very large structure and does a large number of successive transformations on it (typically in the field of NLP (see FLTK, for instance)), then the set of intermediary results can cause excessive usage of memory.
Exemple of code where a "lazy operator" (implemented as a channel) gets transformed into an array by the current goplus handling of list comprehension
import "fmt"
type Counter struct {
Value int
HasNext bool
}
func (c *Counter) Next() {
if c.Value <10 {
c.Value++
c.HasNext = true
} else {
c.HasNext = false
}
}
func GeneratortoIterator(c Counter) chan int {
ch := make(chan int)
go func() {
defer close(ch)
for c.HasNext {
ch <- c.Value
c.Next()
}
}()
return ch
}
func main() {
c := Counter{0, true}
it := GeneratortoIterator(c)
// it is a channel (lazy iteraor)
v1 := [x for x <- it]
// v1 is an array at this point
fmt.Printf("Type: v1 %T\n", v1)
for k, v <- v1 {
println "v[", k, "] = ", v
}
v2 := [x * 2 for x <- v1]
println "v2 = ", v2
}
Workarounds
One way to code it consists in substituting arrays with channels, as in my example, above.
Proposal
Create a second syntax to handle data coming from streams (files, web) as lazy operators:
() vs [] as in python
Example:
A
is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.However if one uses:
Background
In Python if one uses the structure:
A
is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.However if one uses:
Then A is a mere iterator and the values will only be computed as needed.
With the array notation "[]", in a cascade of list comprehensions, all the intermediary results will therefore be kept in RAM.
If one works with a very large structure and does a large number of successive transformations on it (typically in the field of NLP (see FLTK, for instance)), then the set of intermediary results can cause excessive usage of memory.
Exemple of code where a "lazy operator" (implemented as a channel) gets transformed into an array by the current goplus handling of list comprehension
Workarounds
One way to code it consists in substituting arrays with channels, as in my example, above.