Closed vinceecws closed 3 years ago
Assuming saved path="/base/path" & partition_cols=Seq("year", "month"), to read the Parquet for September 2020:
df = spark.sqlContext .read .option("basePath", "/base/path") .parquet("/base/path/year=2020/fetch_month=9")
Similarly, to read the entire Parquet:
df = spark.sqlContext .read .option("basePath", "/base/path") .parquet("/base/path/")
Assuming saved path="/base/path" & partition_cols=Seq("year", "month"), to read the Parquet for September 2020:
Similarly, to read the entire Parquet: