search

coiled / benchmarks

BSD 3-Clause "New" or "Revised" License
28 stars 17 forks source link

Fix memory issues for Spark #1478

Closed hendrikmakait closed 6 months ago

hendrikmakait commented 6 months ago

Fixes OOM problems that led to inconsistent TPC-H benchmarks.

ntabris commented 6 months ago

Maybe it's coincidence that those are the numbers I used as an example, but just to be clear, I have no idea if those are good memory limit numbers.

hendrikmakait commented 6 months ago

Maybe it's coincidence that those are the numbers I used as an example, but just to be clear, I have no idea if those are good memory limit numbers.

Neither do I, but they make intuitive sense to me. Keep a 10% buffer between full memory utilization of a worker and the machine going OOM, and keep a bit more than 10% of that memory for the 10% of Spark overhead that's set per default.

For context, I've run this twice, and tests have passed without a hiccup, so these values are promising at the least.