search

hellonarrativ / spectrify

Export Redshift data and convert to Parquet for use with Redshift Spectrum or other data warehouses.
https://aws.amazon.com/blogs/big-data/narrativ-is-helping-producers-monetize-their-digital-content-with-amazon-redshift/
MIT License
116 stars 25 forks source link

ValueError: Type <class 'sqlalchemy.sql.sqltypes.REAL'> not currently supported by Spectrify. Open an issue? #37

Closed dorianbg closed 5 years ago

dorianbg commented 6 years ago

Description

Copying a table with real column - actually a float4 column

What I Did

spectrify transform

I think it is related to this piece of code https://github.com/hellonarrativ/spectrify/blob/master/spectrify/utils/schema.py

andrewgross commented 6 years ago

I wonder if this is related to Redshift seeing FLOAT as FLOAT8, and everything else seeing FLOAT as FLOAT4, which correlates with REAL.

dorianbg commented 6 years ago

Yes it might be, the only numeric datatypes I have are int4 and float4

dorianbg commented 6 years ago

Seems like altering float4 columns to float8 would be the quickest fix? Though I think this would be mapped to a double in parquet.

andrewgross commented 6 years ago

Correct.

c-nichols commented 6 years ago

Open to PRs for this, if anyone is up for it!

eliran-lightricks commented 5 years ago

Opened #41 !

c-nichols commented 5 years ago

Thanks @eliran-lightricks!!