search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
227 stars 73 forks source link

from_avro converts `\uFFFD` to a question mark #322

Closed EstellaW94 closed 1 year ago

EstellaW94 commented 1 year ago

Hi, we have a scala project leveraing from_avro function, however, when records contains \uFFFD, abris would convert it to a question mark instead of throwing any error, and the question mark is causing issues when we export the record to Teradata. Is there any document for special character handling? how should i do it?

cerveada commented 1 year ago

I don't think Abris changes any specific characters like that.

A question mark is normally used for this character, see: https://codepoints.net/U+FFFD?lang=en

Are you sure you are not just seeing \uFFFD as ?, because some editors display it as that?