search

segmentio / parquet-go

Go library to read/write Parquet files
https://pkg.go.dev/github.com/segmentio/parquet-go
Apache License 2.0
341 stars 104 forks source link

Optimize page buffer allocations #304

Closed achille-roussel closed 2 years ago

achille-roussel commented 2 years ago

This PR addresses the performance regressions from #297.

I explored various solutions and landed on using reference counted buffers to help manage resources internally. I explored exposing a new API to customize memory management but overall it seemed that starting with an internal-only solution at first would be lower risk.

Overall, some benchmarks are a bit slower, some a bit faster, here is a comparison to the main branch:

name                                                     old time/op  new time/op  delta
MergeFiles/BOOLEAN/groups=2,rows=20000                   17.0µs ± 0%  17.4µs ± 0%   +2.20%  (p=0.000 n=8+10)
MergeFiles/INT32/groups=2,rows=20000                     7.05µs ± 2%  7.47µs ± 1%   +5.90%  (p=0.000 n=10+10)
MergeFiles/INT64/groups=2,rows=20000                     7.57µs ± 1%  8.01µs ± 1%   +5.80%  (p=0.000 n=10+10)
MergeFiles/INT96/groups=2,rows=20000                     28.4µs ± 2%  34.4µs ± 2%  +20.90%  (p=0.000 n=10+10)
MergeFiles/FLOAT/groups=2,rows=20000                     6.46µs ± 1%  6.60µs ± 0%   +2.23%  (p=0.000 n=10+9)
MergeFiles/DOUBLE/groups=2,rows=20000                    6.81µs ± 1%  7.06µs ± 0%   +3.70%  (p=0.000 n=10+10)
MergeFiles/BYTE_ARRAY/groups=2,rows=20000                52.1µs ± 0%  47.1µs ± 2%   -9.58%  (p=0.000 n=8+10)
MergeFiles/FIXED_LEN_BYTE_ARRAY/groups=2,rows=20000      34.8µs ± 1%  31.4µs ± 3%   -9.89%  (p=0.000 n=9+10)
MergeFiles/STRING/groups=2,rows=20000                    24.6µs ± 0%  21.2µs ± 2%  -13.81%  (p=0.000 n=10+10)
MergeFiles/STRING_(dict)/groups=2,rows=20000             4.92µs ± 0%  7.93µs ± 2%  +61.18%  (p=0.000 n=9+10)
MergeFiles/UUID/groups=2,rows=20000                      16.4µs ± 0%  16.9µs ± 0%   +3.05%  (p=0.000 n=9+10)
MergeFiles/DECIMAL/groups=2,rows=20000                   6.88µs ± 1%  7.03µs ± 1%   +2.16%  (p=0.000 n=10+9)
MergeFiles/AddressBook/groups=2,rows=20000                774µs ± 1%   809µs ± 1%   +4.57%  (p=0.000 n=10+10)
MergeFiles/one_optional_level/groups=2,rows=20000        22.5µs ± 1%  22.9µs ± 1%   +1.96%  (p=0.000 n=10+9)
MergeFiles/one_repeated_level/groups=2,rows=20000         179µs ± 2%   183µs ± 1%   +2.11%  (p=0.000 n=9+10)
MergeFiles/two_repeated_levels/groups=2,rows=20000        973µs ± 0%   993µs ± 1%   +2.00%  (p=0.000 n=8+10)
MergeFiles/three_repeated_levels/groups=2,rows=20000      973µs ± 1%   992µs ± 0%   +1.97%  (p=0.000 n=9+8)
MergeFiles/nested_lists/groups=2,rows=20000              1.16ms ± 1%  1.18ms ± 1%   +1.23%  (p=0.000 n=9+10)
MergeFiles/key-value_pairs/groups=2,rows=20000            307µs ± 1%   301µs ± 1%   -1.76%  (p=0.000 n=10+9)
MergeFiles/multiple_key-value_pairs/groups=2,rows=20000  1.01ms ± 1%  1.10ms ± 1%   +8.22%  (p=0.000 n=10+10)
MergeFiles/repeated_key-value_pairs/groups=2,rows=20000  1.55ms ± 2%  1.61ms ± 1%   +4.03%  (p=0.000 n=10+9)
MergeFiles/map_of_repeated_values/groups=2,rows=20000     890µs ± 2%   930µs ± 1%   +4.44%  (p=0.000 n=10+10)

name                                                     old row/s    new row/s    delta
MergeFiles/BOOLEAN/groups=2,rows=20000                    56.9M ± 0%   55.7M ± 0%   -2.16%  (p=0.000 n=8+10)
MergeFiles/INT32/groups=2,rows=20000                       137M ± 2%    130M ± 1%   -5.58%  (p=0.000 n=10+10)
MergeFiles/INT64/groups=2,rows=20000                       128M ± 1%    121M ± 1%   -5.48%  (p=0.000 n=10+10)
MergeFiles/INT96/groups=2,rows=20000                      34.1M ± 2%   28.2M ± 2%  -17.29%  (p=0.000 n=10+10)
MergeFiles/FLOAT/groups=2,rows=20000                       150M ± 1%    147M ± 0%   -2.18%  (p=0.000 n=10+9)
MergeFiles/DOUBLE/groups=2,rows=20000                      142M ± 1%    137M ± 0%   -3.56%  (p=0.000 n=10+10)
MergeFiles/BYTE_ARRAY/groups=2,rows=20000                 18.6M ± 0%   20.6M ± 2%  +10.61%  (p=0.000 n=8+10)
MergeFiles/FIXED_LEN_BYTE_ARRAY/groups=2,rows=20000       27.8M ± 1%   30.8M ± 2%  +10.99%  (p=0.000 n=9+10)
MergeFiles/STRING/groups=2,rows=20000                     39.4M ± 0%   45.7M ± 2%  +16.03%  (p=0.000 n=10+10)
MergeFiles/STRING_(dict)/groups=2,rows=20000               197M ± 0%    122M ± 2%  -37.95%  (p=0.000 n=9+10)
MergeFiles/UUID/groups=2,rows=20000                       59.1M ± 0%   57.3M ± 0%   -2.96%  (p=0.000 n=9+10)
MergeFiles/DECIMAL/groups=2,rows=20000                     141M ± 1%    138M ± 1%   -2.12%  (p=0.000 n=10+9)
MergeFiles/AddressBook/groups=2,rows=20000                1.25M ± 1%   1.20M ± 1%   -4.37%  (p=0.000 n=10+10)
MergeFiles/one_optional_level/groups=2,rows=20000         43.0M ± 1%   42.2M ± 1%   -1.92%  (p=0.000 n=10+9)
MergeFiles/one_repeated_level/groups=2,rows=20000         5.59M ± 2%   5.47M ± 1%   -2.07%  (p=0.000 n=9+10)
MergeFiles/two_repeated_levels/groups=2,rows=20000        1.03M ± 0%   1.01M ± 1%   -1.96%  (p=0.000 n=8+10)
MergeFiles/three_repeated_levels/groups=2,rows=20000      1.03M ± 1%   1.01M ± 0%   -1.93%  (p=0.000 n=9+8)
MergeFiles/nested_lists/groups=2,rows=20000                861k ± 1%    851k ± 1%   -1.21%  (p=0.000 n=9+10)
MergeFiles/key-value_pairs/groups=2,rows=20000            3.26M ± 1%   3.32M ± 1%   +1.79%  (p=0.000 n=10+9)
MergeFiles/multiple_key-value_pairs/groups=2,rows=20000    988k ± 1%    913k ± 1%   -7.58%  (p=0.000 n=10+10)
MergeFiles/repeated_key-value_pairs/groups=2,rows=20000    647k ± 2%    622k ± 1%   -3.89%  (p=0.000 n=10+9)
MergeFiles/map_of_repeated_values/groups=2,rows=20000     1.12M ± 2%   1.08M ± 1%   -4.27%  (p=0.000 n=10+10)