manojkarthick / pqrs

Command line tool for inspecting Parquet files
Apache License 2.0
283 stars 27 forks source link

read and write continously #49

Closed txdv closed 5 months ago

txdv commented 6 months ago

The current behaviour is to read all data, buffer it in memory and start writing. I changed it to read a file and write it immediately once read, deallocate the buffered memory. Then move to the next file, here is an example:

image image

You can see that the it also takes less time: 25s compared to 17s because and saves memory, since not everything needs to be buffered in memory