search

snuspl / nemo

Nemo: A flexible data processing system
https://snuspl.github.io/nemo/
Apache License 2.0
22 stars 6 forks source link

Fix DataSkewRuntimePass to play well with compression #769

Closed seojangho closed 6 years ago

seojangho commented 6 years ago

DataSkewRuntimePass balances data distribution based on partition size. Compression reduces partition size, and I've noticed DataSkewRuntimePass conflicts with CompressionPass. (Program executes well, but it gives wrong result.) We need a fix on DataSkewRuntimePass.