search

hellonarrativ / spectrify

Export Redshift data and convert to Parquet for use with Redshift Spectrum or other data warehouses.
https://aws.amazon.com/blogs/big-data/narrativ-is-helping-producers-monetize-their-digital-content-with-amazon-redshift/
MIT License
116 stars 25 forks source link

TEXT fields not supported #51

Closed anomitra closed 5 years ago

anomitra commented 5 years ago

Description

It appears that TEXT fields are not supported (yet). I ran into this while converting a table which has some free text fields.

What I Did

I ran the convert command. The output follows.

ValueError: Type <class 'sqlalchemy.sql.sqltypes.TEXT'> not currently supported by Spectrify.

Is there any particular reason why this was never supported, and can it be taken up as an enhancement? I might take a stab at it, if required.

c-nichols commented 5 years ago

There is no particular reason, it would be greatly appreciated as an enhancement! #41 might be a good PR for inspiration.

anomitra commented 5 years ago

As per the Redshift documentation, a TEXT field is basically just VARCHAR(256). Also, I converted all my TEXT fields to VARCHAR fields and everything worked fine.

It seems like adding a mapping for TEXT in the data type mappings should suffice, along with a few regression tests. I'll try and get around to it during the weekend if I can!

c-nichols commented 5 years ago

@anomitra thank you for this!