Myria broadcast partitioning support

uwescience / raco

Compilation and rule-based optimization framework for relational algebra. Raco is the language, optimization, and query translation layer for the Myria project.

Other

72 stars 19 forks source link

Myria broadcast partitioning support #548

Closed senderista closed 7 years ago

senderista commented 7 years ago

This supersedes https://github.com/uwescience/raco/pull/517. I also fixed another issue in this PR (https://github.com/uwescience/raco/issues/495) since it needed to be done to properly support debroadcast of QueryScans.

coveralls commented 7 years ago

Coverage decreased (-0.09%) to 91.858% when pulling d9e22cf5ebf9ee413ca65791a9242d5e2adaa5d5 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

coveralls commented 7 years ago

Coverage decreased (-0.09%) to 91.858% when pulling e012d4d243e75afdf65492f5a37a1f22345a057d on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

shrjain commented 7 years ago

Mostly nitpicks, otherwise looks good.

coveralls commented 7 years ago

Coverage decreased (-0.09%) to 91.852% when pulling 837c2973c26807d55025d33259ede0bd562e66c5 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

coveralls commented 7 years ago

Coverage decreased (-0.09%) to 91.852% when pulling 4b9e2bf67a5a036133281643aea4ed4460273f6e on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

senderista commented 7 years ago

@shrjain if you wanna take a look at the new ShuffleAfterFileScan rule, that would be swell :)

shrjain commented 7 years ago

@senderista: Done. Looks good.

shrjain commented 7 years ago

@senderista: But from the mail thread I gather that this won't work without pushselects rule being updated.

coveralls commented 7 years ago

Coverage decreased (-0.05%) to 91.892% when pulling 275e2c0053b35e79076e0c2e4b7ff9060c055288 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

jingjingwang commented 7 years ago

Will the push-select rule be updated in this PR?

senderista commented 7 years ago

I guess we may as well implement it, it shouldn't be hard...

coveralls commented 7 years ago

Coverage decreased (-0.06%) to 91.882% when pulling fb6f819a8b7d25444737085a9da0808bc39f7287 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

coveralls commented 7 years ago

Coverage increased (+0.01%) to 91.956% when pulling 2a1633f0981439a3c980737305c38aa430819159 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

senderista commented 7 years ago

@jingjingwang @shrjain I think this should be ready to merge if you can make a final pass. I've added the new PushSelectThroughShuffle rule and some minimal test coverage for the new rules (ShuffleAfterFileScan, PushSelectThroughShuffle).

coveralls commented 7 years ago

Coverage increased (+0.02%) to 91.963% when pulling b1c052d1039184ba482ca2b5533310072f682183 on myria_broadcast into 8f8e38f6d5e870bc0b9098d0dff5316c6b44e42a on master.

bmyerz commented 7 years ago

@senderista Glad to see this make it in!! Sorry I couldn't help push it through.