search

segmentio / parquet-go

Go library to read/write Parquet files
https://pkg.go.dev/github.com/segmentio/parquet-go
Apache License 2.0
341 stars 58 forks source link

set byte slice capacity when growing output buffers #386

Closed achille-roussel closed 2 years ago

achille-roussel commented 2 years ago

I have a theory that #372 might be caused by growing memory allocations in small increments due to calling make with only the slice size and leaving the capacity allocation up to the runtime.

This PR modifies the encoding packages to always create slices with twice the capacity as the target size to ensure that we have room for growth when the memory areas are reused.

@mdisibio let me know if the change helps address the issue that you originally reported!