As @cwwalter mentioned, having _common_metadata and _metadata in the parquet file folder can potentially speed up load time for dask and spark.
We should be able to add the steps to generate these metadata files in write_gcr_to_parquet.py. Alternatively, we can write a post-processing script to generate those files (could be useful for existing datasets).
As @cwwalter mentioned, having
_common_metadataand_metadatain the parquet file folder can potentially speed up load time for dask and spark.We should be able to add the steps to generate these metadata files in
write_gcr_to_parquet.py. Alternatively, we can write a post-processing script to generate those files (could be useful for existing datasets).Ref: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html