Any way to Read parquet records before appending to file

[govthamreddy]

Is there anyway to read the parquet converted record before writing it into parquet file. My requirement is to create a parquet file directly into azure data lake without storing it locally.

[dobesv]

Using the parquets package I've been able to stream out parquet data without saving to a temporary file, it supports node streams. It will write out the data one row group at a time, though, so you do need enough memory to hold a full row group. This is a limitation of parquet.

[vishald2509]

Using the parquets package I've been able to stream out parquet data without saving to a temporary file, it supports node streams. It will write out the data one row group at a time, though, so you do need enough memory to hold a full row group. This is a limitation of parquet.

can you please provide more insight into the code perspective? It will be very helpful.