search

databricks / spark-avro

Avro Data Source for Apache Spark
http://databricks.com/
Apache License 2.0
539 stars 310 forks source link

Consume Avro stream from Kafka topic #232

Open ananth3010 opened 7 years ago

ananth3010 commented 7 years ago

Hi

The current version reads .avro files from HDFS (or) any FS file path. I am storing my avro file stream in Kafka. Do you have a utility to read a avro stream (or) file from a Kafka topic?

If so could you let me know?

ananth3010 commented 7 years ago

216 has this resolved. Is there a documentation on how to use this?

ananth3010 commented 7 years ago

201 confirms this. Pasting it here for others benefit:

You can do this with DStream.foreachRDD { rdd => df = rddToDataFrame(rdd) }

OneCricketeer commented 5 years ago

This post shows Spark 2.4 Avro features and Kafka https://databricks.com/blog/2018/11/30/apache-avro-as-a-built-in-data-source-in-apache-spark-2-4.html