hellonarrativ / spectrify

Export Redshift data and convert to Parquet for use with Redshift Spectrum or other data warehouses.
https://aws.amazon.com/blogs/big-data/narrativ-is-helping-producers-monetize-their-digital-content-with-amazon-redshift/
MIT License
116 stars 25 forks source link

Type issue with date. #19

Closed jarrettj closed 6 years ago

jarrettj commented 6 years ago

Description

Use convert command.

What I Did

spectrify --host=blah.redshift.amazonaws.com --user=blah --db=blah convert tableblah 's3://blah'

/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/spectrify", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/spectrify/main.py", line 59, in convert
    sa_table = get_table_schema(engine, table)
  File "/usr/local/lib/python2.7/dist-packages/spectrify/utils/schema.py", line 52, in get_table_schema
    col.type.__class__
ValueError: Type <class 'sqlalchemy.sql.sqltypes.DATE'> not currently supported by Spectrify. Open an issue?
c-nichols commented 6 years ago

Hi @jarrettj, thanks for the feedback. Good news -- support for the date column has already been added on master branch. I'm hoping to cut a new release within the next couple days -- I'm waiting to see if I can get DECIMAL/NUMERIC support pulled together.

jarrettj commented 6 years ago

Awesome! Done just like that! I'll be patiently waiting. Thanks man!

Regards JJ

c-nichols commented 6 years ago

released!

jarrettj commented 6 years ago

Awesome! Thanks.