flipkart-incubator / databuilderframework

A data driven execution engine
33 stars 29 forks source link

Builders of the same rank terminate when first one does not run #7

Closed gokulvanan closed 8 years ago

gokulvanan commented 8 years ago

Since Builder is topo sorted, there is a check builder executor to break if the first builder in the respective rank does not run. But this a bug as there are cases where builders in the same rank can consume independent data such that first builder will expect Data A which is produced by builder above it but second builder on the other hand does not depend on Data A.
The reason these two builders are in the same rank is because when execution graph is builder from bottom up.

santanusinha commented 8 years ago

Pass a Strategy object from above to handle exceptions and whether to eat them up or stop when it encounters one.

gokulvanan commented 8 years ago

Let me come back with a test case to illustrate this better. I have a bigger question regarding ranking logic with this.. Will explain with that test cases.

gokulvanan commented 8 years ago

Check the pastebin link. This is the ExecutionGraph generated by a new complex added in test-cases branch in my fork of databuilderframework. This describes some of the real world cases we see here with builder name abstracted out as alphabets A to K.

http://pastebin.com/RMeKp6jh

In this case, Builders A1, A2, A3 could be in the same rank. But they are not.. This is because we build bottom up. But with application on optionals, this design was expecting either A1 or A2 to generate data and for the case of A1 generating data there are a whole lot of builders down the line B1 to B4 which run and then go to builder C, whereas A2 goes directly to builder C.

Here existing logic to break the loop on same rank builders when the first one does not run is a problem. As Builder B1, B2 and A2 are ranked 8 and for a case when A2 should run where A1 returns null. B1 will not run as it depends on A1 output and the for loop breaks here. Hence A2 is never run.

Hope this description is adequate else we can connect offline on this