Closed ryan6310 closed 1 month ago
@ryan6310 Which version of the connector are you using? Complex type support was added to version 6.0.0 and later.
I am using EMR Serverless 6.12.0.
@ryan6310, EMR 6.12 uses an older version of the connector (5.1.0). Please upgrade to EMR 7.0 or newer to utilize a post 6.0.0 version of the connector.
I am working on a data pipeline that involves multiple steps of reading and writings to redshift. I am having trouble with arrays of strings that are getting written to super type.
In spark the schema is
and in spark I see what I expect
This is then written to redshift. The column metadata contain
redshift_type = Super
In redshift I can see the resulting items in the form
When I load the table from redshift and set the schema I get a parsing error
Is this the expected behavior? Do I need to manually parse the loaded string into arrays?
Full trace below: