search

aphyr / tesser

Clojure reducers, but for parallel execution: locally and on distributed systems.
867 stars 39 forks source link

Stack overflow error when too few chunks #16

Closed neverfox closed 8 years ago

neverfox commented 8 years ago

I have noticed that I will get stack overflow errors if I don't set the number of reducible chunks high enough when operating on large collections. The stack is a repeat of the following (my combiner is concat):

core.clj:  137  clojure.core/seq
core.clj:  717  clojure.core/concat/fn
LazySeq.java:   40  clojure.lang.LazySeq/sval
LazySeq.java:   49  clojure.lang.LazySeq/seq
RT.java:  525  clojure.lang.RT/seq

Thus far, there is always a number of chunks that allow the fold to complete successfully, but it's a guess as to what it needs to be. Is this problem something you recognize? If so, any idea why it happens? Is there a rule of thumb for setting the number of chunks?

neverfox commented 8 years ago

Could it be this? It's the same error.

neverfox commented 8 years ago

Pretty sure that's the problem. I'm replacing my combiner with into as suggested. Hopefully someone else finds this useful.