search

h2oai / db-benchmark

reproducible benchmark of database-like ops
https://h2oai.github.io/db-benchmark
Mozilla Public License 2.0
325 stars 88 forks source link

enable multithreading in Julia #196

Closed bkamins closed 3 years ago

bkamins commented 3 years ago

As discussed in https://github.com/h2oai/db-benchmark/pull/194 I open a separate PR enabling multi-threading support. I assume that -S option works on your test system.

Thank you!

jangorecki commented 3 years ago

How can I print from julia how many cores are being used? so I can test this change

Machine has 40 cores, do you want to use only 20?

bkamins commented 3 years ago

use Threads.nthreads() to get the number of cores Julia process is started with.

Regarding 20 vs 40 I was not sure about the architecture of the machine. I would start with 20 (which I assume is the number of physical cores). We are still working on multi-threading code after 1.0 release (1.0 was mostly API stabilization, so things are not polished there yet - as I am sure you know these issues are hard and quite hardware dependent)

bkamins commented 3 years ago

@jangorecki - your tests helped uncover a bug in our CSV reader (which kills the performance), so it is super useful to have them 😄.

We will let you know how things are resolved and in this PR I will do appropriate adjustments to CSV reader settings when we are done.

CC @nalimilan