search

rchillyard / Majabigwaduce

A framework to allow MapReduce applications to use Akka actors
12 stars 4 forks source link

Join works only when key never changed #7

Closed rchillyard closed 3 years ago

rchillyard commented 5 years ago

See DataDefinitionSpec

rchillyard commented 5 years ago

Note two occurrences in that Spec file

vinit2107 commented 3 years ago

Two issues causing this problem. Issue 1: The values were stored in a HashMap, and upon updating the keys the order was not maintained. This was resolved by updating mutable.HashMap with mutable.LinkedHashMap. Issue 2: The join happened on original keys rather than the updated keys. This caused the empty map to be sent to the master. This was resolved by creating a new function "joinMap2" in DataDefinition.scala which transforms the keys, joins on the transformed keys but maintains the original key in the resultant LazyDD. This was done because the key transformation takes place at the end when the apply method calls the evaluate method in BaseDD.

rchillyard commented 3 years ago

Excellent work, Vinit! Thank you so much :)