search

h2oai / db-benchmark

reproducible benchmark of database-like ops
https://h2oai.github.io/db-benchmark
Mozilla Public License 2.0
321 stars 85 forks source link

Polars [Rust native solution] #185

Open ritchie46 opened 3 years ago

ritchie46 commented 3 years ago

This PR is the rust native solution of Polars. I renamed the python folder for the time being. Maybe it can be removed if all works?

I now log the duration of the queries (in milliseconds) and the chk answers to stdout. What kind of format is best for you?

ritchie46 commented 3 years ago

Hi @jangorecki, Is there anything I can do on this?

jangorecki commented 3 years ago

@ritchie46 be patient. Unfortunately I have two other projects in queue as of now.

ritchie46 commented 3 years ago

Ok, I can do that. :)

Dandandan commented 3 years ago

@jangorecki what would be needed to get rust native benchmarks in here?

jangorecki commented 3 years ago

Design well how to handle compiled languages. We don't have any yet in this benchmark suite. Maybe it is already in this PR made well enough. It needs time to analyze. As long as python client is giving a minimal overhead that doesn't grow with data size, then it is not that high priority.

ritchie46 commented 3 years ago

I think the most notable difference is the conservative compiling we have to do for distributing python libraries. The compiler doesn't have the query in scope and cannot utilize all SIMD instructions available on the target architecture. At the moment only Julia compiles on the native machine JIT, or (just ahead of time) if I am not mistaken.