search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
229 stars 75 forks source link

how can we write data frame to multiple avro Kafka topics ? #32

Closed abhisam closed 5 years ago

abhisam commented 5 years ago

i want to write dataframe to multiple topics

felipemmelo commented 5 years ago

Hello @abhisam ,

To make sure I understood your question, would you like to register the schema for the same dataframe into multiple Schema Registry topics and then dispatch that dataframe to those topics?

Cheers.

abhisam commented 5 years ago

Hi @felipemmelo , i want to write a data frame to multiple topic based on conditions. can we write that in single write stream ?

felipemmelo commented 5 years ago

Hi @abhisam , I'm not sure structure streams allow it. From the library perspective there should be no problem, since it only encodes the payload for you. I'll check if Spark itself allows it.

Also, if I'm not mistaken, since Spark 2.4 there is a new API called foreachBatch or something like this. Have you tried any of these options without ABRiS, to see if it is allowed by Spark?

abhisam commented 5 years ago

i am using spark 2.3. i didn't try spark 2.4 . is it possible to do in 2.3?

felipemmelo commented 5 years ago

I don't think it is possible to do this using Structured Streaming. Also, I'm happy to help, however, I'll close this issue since it is not directly related to this project, ok?