search

aphyr / tesser

Clojure reducers, but for parallel execution: locally and on distributed systems.
873 stars 39 forks source link

Fix stack overflow issue when using t/into #18

Closed leon-barrett closed 8 years ago

leon-barrett commented 8 years ago

t/into used concat in a way that would cause stack overflows with large numbers of chunks.

leon-barrett commented 8 years ago

I ran into this StackOverflowError when I was using into. I think this is a simple, efficient fix.

I can see that using a list for accumulation might be more efficient than using a vector, but it would reverse outputs, and that might surprise users. Using a vector is probably efficient enough to conj to, right?

aphyr commented 8 years ago

Thanks Leon. :)

aphyr commented 8 years ago

Er, this patch breaks the tests. Reverting; I'll see if I can find a different solution.

leon-barrett commented 8 years ago

Oh, really? It passed lein test for me. What fails?

aphyr commented 8 years ago

group-by-spec explodes for some reason.

leon-barrett commented 8 years ago

I think group-by doesn't quite fulfill the contract--it assumes that each input works as a combiner-identity. I have another change to fix it. Please stand by.

aphyr commented 8 years ago

Fixed! See de2986f

leon-barrett commented 8 years ago

Ah, just so, nicely. edit: Thanks!