search

datafusion-contrib / datafusion-catalogprovider-glue

Apache License 2.0
21 stars 4 forks source link

Support `TextInputFormat` #8

Open matthewmturner opened 2 years ago

matthewmturner commented 2 years ago

I have JSON tables in my glue catalog and I get the following error for those:

No support for: org.apache.hadoop.mapred.TextInputFormat, org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

timvw commented 2 years ago

Might also want to specify the Serde(s) for json

-> Eg: CSV is "org.apache.hadoop.mapred.TextInputFormat", "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",

https://docs.aws.amazon.com/athena/latest/ug/json-serde.html

-> https://github.com/datafusion-contrib/datafusion-catalogprovider-glue/compare/main...support-json