wait for all broadcast being cleaned after each benchmark

databricks / spark-sql-perf

Apache License 2.0

582 stars 406 forks source link

wait for all broadcast being cleaned after each benchmark #170

Closed cloud-fan closed 5 years ago

cloud-fan commented 5 years ago

The broadcasted data stay in the executor block manager and make executor have less execution memory. We should wait it to be cleaner before running next benchmark

cloud-fan commented 5 years ago

@npoggi do we have jenkins for this project?

npoggi commented 5 years ago

@cloud-fan we use travis-ci for this project. Why dit you closed this PR?

cloud-fan commented 5 years ago

For the perf regression, the problem is task skew not broadcast, so this patch is not useful now.