sunchao / parquet-rs

Apache Parquet implementation in Rust
Apache License 2.0
149 stars 20 forks source link

add two benchmark test with new data file #155

Closed asdetrefle closed 6 years ago

asdetrefle commented 6 years ago

The second benchmark test is a for low level column reader call on the new data file. This is theoretically the lower bound of read time needed. We also should aim for this when we implement Column and Table. Now we can see huge potential for the current implementation:

test record_reader_10k_collect   ... bench:  17,203,101 ns/iter (+/- 109,298) = 38 MB/s
test record_reader_stock_collect ... bench: 116,401,996 ns/iter (+/- 1,467,345) = 11 MB/s
test record_reader_stock_column  ... bench:   6,892,490 ns/iter (+/- 26,985) = 187 MB/s
coveralls commented 6 years ago

Pull Request Test Coverage Report for Build 607


Files with Coverage Reduction New Missed Lines %
encodings/encoding.rs 1 94.82%
<!-- Total: 1 -->
Totals Coverage Status
Change from base Build 602: -0.008%
Covered Lines: 12049
Relevant Lines: 12618

💛 - Coveralls
sadikovi commented 6 years ago

Thanks for the benchmarks! Now we need to figure out what we are going to do to improve the performance.

sadikovi commented 6 years ago

LGTM

sunchao commented 6 years ago

LGTM too. Thanks @asdetrefle !

@sadikovi : I think you should be able to merge this. Can you try it?

sadikovi commented 6 years ago

@sunchao Yes, I can merge as well! Thanks for the review and sign off. Now, I can help with merging PRs too. @asdetrefle - great job!