spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
137 stars 63 forks source link

Add 'include_column_list' parameter #58

Closed eeshugerman closed 4 years ago

eeshugerman commented 4 years ago

I found this un-merged patch by @kyrozetera on the old databricks repo (link), and it solves an issue I'm facing.

My use case

Many of the Redshift tables that I'm working with have created_dt columns, like so: created_dt timestamp not null default current_time These columns are intended to be left unspecified on inserts/copies, so that the default is used. But for this to work, a column list must be included in the COPY statement, or else I get an error: Missing data for not-null field

Author's use case

It appears the author's motivation for this patch is related to, but different than my own: https://github.com/databricks/spark-redshift/issues/340

codecov-io commented 4 years ago

Codecov Report

Merging #58 into master will increase coverage by 0.16%. The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #58      +/-   ##
==========================================
+ Coverage   73.92%   74.09%   +0.16%     
==========================================
  Files          15       15              
  Lines         767      772       +5     
  Branches      103      106       +3     
==========================================
+ Hits          567      572       +5     
  Misses        200      200
eeshugerman commented 4 years ago

Thanks for reviewing @lucagiovagnoli! I've made the requested changes to the tests.

  1. Bump version and changelog (patch version should be ok as this is backwards compatible)

I bumped the minor version instead, since this PR adds functionality. From https://semver.org:

Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards compatible manner, and PATCH version when you make backwards compatible bug fixes.

But I can switch to patch if you prefer.