search

memsql / singlestore-spark-connector

A connector for SingleStore and Spark
Apache License 2.0
160 stars 54 forks source link

avro Date format exception #62

Closed bolcman closed 4 years ago

bolcman commented 4 years ago

When I switch to loadDataFormat=Avro

I get an error when I have DataFrame with Date column

org.apache.avro.AvroRuntimeException: Unknown datum type java.sql.Date:

thanks, Aleks

AdalbertMemSQL commented 4 years ago

Hello, Alex.

The Avro format does not support the writing of TIMESTAMP and DATE types. As a result, the SingleStore Spark Connector currently does not support these types with Avro serialization. https://docs.singlestore.com/v7.1/third-party-integrations/spark-3-connector/#data-type-conversion-remarks

blinov-ivan commented 4 years ago

@bolcman is there anything else we could help you with? To write Date type please use CSV format instead of Avro

blinov-ivan commented 4 years ago

@bolcman closing this issue for now - please reopen if you will need some additional help with Avro serialization.