search

move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

[Feature/Addition] parsons.databases.redshift.copy() should infer column types #941

Closed austinweisgrau closed 7 months ago

austinweisgrau commented 7 months ago

By default, parsons.databases.redshift.copy() makes all column types VARCHAR in redshift. It is possible to pass a columntypes dictionary mapping argument to explicitly set column types, but this method should be able to infer column types from the parsons.Table input if no columntypes are explicitly provided.

This behavior is built in to the new parsons.google.bigquery.BigQuery.copy() method (see here) and should be available for copying into redshift as well.

Detailed Description

Context

Creating all column types as VARCHAR is more naive than is necessary, the source parsons Table has information available about column types and those should be used to create column types in Redshift as well.

Possible Implementation

Should be able to closely match the behavior of this method for bigquery, with changes for Redshift compatibility.

Priority

austinweisgrau commented 7 months ago

Actually this behavior is supposed to exist with parsons.databases.redshift.rs_create_table.generate_data_types(), investigating why its not working for me in a certain instance...

austinweisgrau commented 7 months ago

Closing this issue in favor of the more precise https://github.com/move-coop/parsons/issues/942