Previously we reduced overall memory consumption of parquet-go by leveling our buffer pools. This PR collapses all buffer pools into 1 to reduce memory consumption even further.
I believe that they were broken out into individual pools to help consolidate buffers of similar sizes in different pools. However, with the leveling change sharing pools should reduce overall consumption. In practice we are seeing roughly 5% reduction in working set on the most heavily impacted processes.
Previously we reduced overall memory consumption of parquet-go by leveling our buffer pools. This PR collapses all buffer pools into 1 to reduce memory consumption even further.
I believe that they were broken out into individual pools to help consolidate buffers of similar sizes in different pools. However, with the leveling change sharing pools should reduce overall consumption. In practice we are seeing roughly 5% reduction in working set on the most heavily impacted processes.