search

segmentio / parquet-go

Go library to read/write Parquet files
https://pkg.go.dev/github.com/segmentio/parquet-go
Apache License 2.0
341 stars 102 forks source link

Value.Bytes() return value directly #508

Closed thorfour closed 1 year ago

thorfour commented 1 year ago

Previously the Value.Bytes() function was calling AppendBytes for it's implementation. This causes uncessary memory to be allocated.

This changes the Bytes function to directly return the values instead of performing an append to nil.

Running a benchmark that read in a large amount of data from Parquet files we see the following improvements

name              old time/op    new time/op    delta
_PreAggregate-10     4.84s ± 1%     4.39s ± 3%   -9.18%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op   delta
_PreAggregate-10    14.6GB ± 0%    10.5GB ± 1%  -27.96%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
_PreAggregate-10      162M ± 0%      128M ± 0%  -21.44%  (p=0.000 n=10+10)