Create Parquet Stream Reader

AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark

Apache License 2.0

44 stars 13 forks source link

Create Parquet Stream Reader #163

Closed kevinwallimann closed 3 years ago

kevinwallimann commented 4 years ago

Structured Streaming supports File sources out of the box. A ParquetStreamReader should be implemented to enable parquet file sources for users of hyperdrive, analogously to the existing ParquetStreamWriter

https://spark.apache.org/docs/2.4.4/structured-streaming-programming-guide.html#input-sources