search

NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

Node getting recomputed #265

Closed espringe closed 11 years ago

espringe commented 11 years ago

Looks like a nodes are sometimes getting recomputed instead of reused, which is a problem for non-deterministic computations.

There's a unit-test here: https://github.com/NICTA/scoobi/pull/264

which currently fails because of this

etorreborre commented 11 years ago

This is actually expected behaviour with the way you wrote the partition example. There is no way for Scoobi to be aware of the whole graph unless you pass it entirely to the persist method. This is why I added persist(l, r) before making the calls to .run. In that case the computation nodes will only be computed once.

espringe commented 11 years ago

ohhhhhhhhhhhhhh, touche. I'm a moron.

On Wed, May 29, 2013 at 3:49 PM, Eric Torreborre notifications@github.comwrote:

This is actually expected behaviour with the way you wrote the partitionexample. There is no way for Scoobi to be aware of the whole graph unless you pass it entirely to the persist method. This is why I added persist(l, r) before making the calls to .run. In that case the computation nodes will only be computed once.

— Reply to this email directly or view it on GitHubhttps://github.com/NICTA/scoobi/issues/265#issuecomment-18651358 .