This PR adds a new method to parquet.Type named EstimateDecodeSize which is similar in functionality to EstimateSize but provides more precise results when working with pages of variable size values (e.g. BYTE_ARRAY).
The use of this new method allows us to better size the output buffer where pages are decode, which greatly reduces memory pressure by avoiding reallocation of the buffer since the one that was selected to decode BYTE_ARRAY pages was almost always too small.
This PR adds a new method to
parquet.TypenamedEstimateDecodeSizewhich is similar in functionality toEstimateSizebut provides more precise results when working with pages of variable size values (e.g. BYTE_ARRAY).The use of this new method allows us to better size the output buffer where pages are decode, which greatly reduces memory pressure by avoiding reallocation of the buffer since the one that was selected to decode BYTE_ARRAY pages was almost always too small.