search

memsql / singlestore-spark-connector

A connector for SingleStore and Spark
Apache License 2.0
160 stars 54 forks source link

cant add JSON field from within DataFrame.withColumn call #22

Closed akonopko closed 4 years ago

akonopko commented 8 years ago

Hello

I have a Map field inside DataFrame which i need to save as JSON to MemSQL. When i do conversion to JsonValue, i got following error

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String

Here is code example which reproduces the problem:

https://gist.github.com/akonopko/495f2ece5398700e000ec63dafb61c2b

I was able to fix this with a workaround:

https://gist.github.com/akonopko/af48c91170fa8f344bcb63aa31b82037

carlsverre commented 8 years ago

Thanks for the bug report akonopko! We will take a good look at this as soon as time permits.

carlsverre commented 4 years ago

The JsonValue feature was removed in the v2 version of the connector. Everything now works as long as you read/write with strings.