ironSource / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format
MIT License
346 stars 175 forks source link

Return array of buffers instead of buffer.concat #54

Open ZJONSSON opened 6 years ago

ZJONSSON commented 6 years ago

Buffer.concat, particularly on the whole rowgroup buffers can get expensive memory wise, as we'll end up with two copies of the whole buffer (the individual parts and the concatenated version) for a moment before the garbage-collector picks up the pieces. This can limit the maximum size of the rowgroup.

It might be more efficient to return bodyParts (a sequential array of bufffers) in general instead of buffer.concat and then in writeSection we loop through the bodyParts and write them sequentially.

ZJONSSON commented 6 years ago

see https://github.com/ZJONSSON/parquetjs/tree/bodyParts for WIP (contingent on https://github.com/ironSource/parquetjs/pull/52)