YotpoLtd / metorikku

A simplified, lightweight ETL Framework based on Apache Spark
https://yotpoltd.github.io/metorikku/
MIT License
583 stars 155 forks source link

Add support for Kafka input in batch mode #456

Open tooptoop4 opened 2 years ago

tooptoop4 commented 2 years ago

Instead of streaming mode, allow reading entire topic into a df at startup. ie https://stackoverflow.com/questions/56701884/how-to-load-all-records-from-kafka-topic-using-spark-in-batch-mode