Open juarezr opened 6 years ago
Documentation says that should set compression in SparkConf:
spark.conf.set("spark.sql.avro.compression.codec", "deflate")
But others formats like parquet allows setting it in DataFrameWriter options:
DataFrameWriter writer = rowDataset.write() .format("com.databricks.spark.avro") .option("compression","snappy") .save(path);
For consistency, spark-avro could also respect this setting.
Documentation says that should set compression in SparkConf:
But others formats like parquet allows setting it in DataFrameWriter options:
For consistency, spark-avro could also respect this setting.