search

MeltanoLabs / target-athena

Singer.io Target for AWS Athena.
Other
5 stars 16 forks source link

target not generating correct column definitions for tap-mongodb #17

Closed andrewcstewart closed 3 years ago

andrewcstewart commented 3 years ago

Testing #12 with the transferwise variant of tap-mongodb, it looks like every single schema generated by the target only contains _id,document, and _sdc_deleted_at. I'm not exactly sure what's going on here, but I'm guessing we'll have to by default flatten self.schema or something?

CREATE EXTERNAL TABLE `patterns`(
  `_id` string COMMENT 'from deserializer', 
  `document` string COMMENT 'from deserializer', 
  `_sdc_deleted_at` string COMMENT 'from deserializer')
ROW FORMAT SERDE 
  'org.openx.data.jsonserde.JsonSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION
  '-'
andrewcstewart commented 3 years ago

I think this may have been due toflatten_fields being applied to records but not reflected in the schema. I made flatten_fields a config option for the time being.