[BUG] ExcelGenerator makeConverter is missing cases for many data types - Githubissues

crealytics / spark-excel

A Spark plugin for reading and writing Excel files

Apache License 2.0

463 stars 147 forks source link

[BUG] ExcelGenerator makeConverter is missing cases for many data types #625

Open pjfanning opened 2 years ago

pjfanning commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

See #623 for one issue (booleans). Some of the main types are supported. Many are not.

Expected Behavior

https://spark.apache.org/docs/latest/sql-ref-datatypes.html

Missing support includes

VarcharType
CharType
BinaryType
Interval types (YearMonthIntervalType, DayTimeIntervalType, etc.)
ArrayType
MapType
StructType/StructField

Steps To Reproduce

Use a dataframe with columns that have less common data types

Environment

- Spark version:
- Spark-Excel version:
- OS:
- Cluster environment

Anything else?

No response

pjfanning commented 2 years ago

Varchar/char types are not fully supported in Spark. I get this when creating a dataframe with Spark 3.2.2.

[info] org.apache.spark.sql.AnalysisException: char/varchar type can only be used in the table schema. You can set spark.sql.legacy.charVarcharAsString to true, so that Spark treat them as string type as same as Spark 3.0 and earlier [info] at org.apache.spark.sql.errors.QueryCompilationErrors$.charOrVarcharTypeAsStringUnsupportedError(QueryCompilationErrors.scala:1599)