search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
135 stars 62 forks source link

Use SMALLINT if the Spark dataType is ShortType #152

Closed aajisaka closed 5 months ago

aajisaka commented 7 months ago

Now the connector creates Redshift table with INTEGER column if the DataFrame column is short. After this change, the connector creates SMALLINT column.

Note that this change will change the behavior if the DataFrame has ShortType column

The issue was opened long time ago https://github.com/databricks/spark-redshift/issues/401 but not fixed

bsharifi commented 7 months ago

@aajisaka Thank you for your feature submission. We are currently reviewing this feature.

bsharifi commented 6 months ago

@aajisaka We have confirmed this is a bug and will address this in an upcoming release.

bsharifi commented 5 months ago

Thank you @aajisaka for submitting a fix for this issue. I am merging this into the mainline and it will be part of the next release.