Make use of parquet format instead of avro

[yu-iskw]

Hi @nevillelyh

We use avro format to save a dataframe to GCS before loading the avro files to bigquery. One of the biggest advantages of avro format is that bigquery can read the schema from the avro metadata. However, avro format doesn't support timestamp type. So we need a twists to store a dataframe which includes timestamp columns.

I guess parquet support timestamp type. Moreover, if bigquery can load parquet files on GCS without explicit schema, it would be better to use parquet format. What do you think?

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet

[nevillelyh]

You could give it a shot. But I'm not sure the timestamp support alone justifies the cost.

• Parquet load support in BQ is not as mature as Avro and we've seen some rough edges.
• BQ supports loading Parquet only and not exporting right? So timestamp may not round trip?
• Parquet read/write uses a lot more memory than Avro, and since we're not doing column projection or row predicate, we're not leveraging any Parquet benefits.

[yu-iskw]

Thank you for the comment. You make good points. We should miss the pros of avro against only the timestamp type benefit. We can also modify spark-avro itself. I think it would be better for our case.

https://github.com/databricks/spark-avro/issues/229