search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
227 stars 73 forks source link

malformed records to topic #345

Closed talperetz1 closed 11 months ago

talperetz1 commented 11 months ago

I have a streams that consumes avro messages from kafka topic perform from_avro by the provided schema and write the dataframe to s3. There is a possibility that in some point a an avro message with incompatible schema will arrive to my topic, when that happens I will get Malformed record. Now I dont wnat to ignore this message and skip to the next and "correct" messages, I want to take that malformed record and write it to a topic/s3 bucket with his "malformed schema". There is a wat to do so? Thank you?

cerveada commented 11 months ago

Error handling is described her: https://github.com/AbsaOSS/ABRiS#de-serialisation-error-handling

If the options available there are not sufficient for your use case, you can implement a custom error handler.

talperetz1 commented 11 months ago

Thank you