A frame consisting of CharType(255) doesn't map to anything in Hive/Parquet sink

[hannesmiller]

The method io.eels.component.parquet.ParquetSchemaFns.toParquetType doesn't handle Frames with a CharType and hence the following exception:

scala.MatchError: CharType(255) (of class io.eels.schema.CharType) at io.eels.component.parquet.ParquetSchemaFns$.toParquetType(ParquetSchemaFns.scala:72) at io.eels.component.parquet.ParquetSchemaFns$$anonfun$2.apply(ParquetSchemaFns.scala:110) at io.eels.component.parquet.ParquetSchemaFns$$anonfun$2.apply(ParquetSchemaFns.scala:110) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at io.eels.component.parquet.ParquetSchemaFns$.toParquetSchema(ParquetSchemaFns.scala:110) at io.eels.component.parquet.ParquetWriterFn$.apply(ParquetWriterFn.scala:30) at io.eels.component.parquet.ParquetSink$$anon$1.(ParquetSink.scala:16) at io.eels.component.parquet.ParquetSink.writer(ParquetSink.scala:14) at io.eels.actions.SinkAction$.execute(SinkAction.scala:10)

• A workaround is to manually map the Frame type to a StringType
• However I think CharType should map to the Parquet logical type of FIXED_LEN_BYTE_ARRAY - I also think the same should be done for VarcharType
• Note varchar(size) and char(size) are valid Hive types and are handled by the Hive Parquet Serde.
• Note Hive Char types are similar to Varchar but they are fixed-length meaning that values shorter than the specified length value are padded with spaces but trailing spaces are not important during comparisons. The maximum length is fixed at 255.

[sksamuel]

I've mapped it to Strings. The reason I've done that rather than FIXED_LEN_BYTE_ARRAY is because if we use FIXED_LEN_BYTE_ARRAY, will hive or impala know to interpret it as a string if we query it? Eel would know, but I'm not sure other libraries would.

[hannesmiller]

Yeah I can live with that 😄