search

vertica / spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Apache License 2.0
20 stars 23 forks source link

added option for truncation in overwrite #509

Closed ai-bq closed 1 year ago

ai-bq commented 1 year ago

Summary

Some users require the option of truncating a table rather than dropping it in overwrite mode to retain table privileges.

Description

Truncate table was added to table utilities as well as the option of truncation through the dataframe writing options.

Related Issue

507

Additional Reviewers

@alexey-temnikov @jeremyp-bq @jonathanl-bq @alexr-bq

CLAassistant commented 1 year ago

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Alex Impert seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

codecov[bot] commented 1 year ago

Codecov Report

Merging #509 (7e1125d) into main (00d0c05) will decrease coverage by 0.15%. The diff coverage is 63.63%.

@@            Coverage Diff             @@
##             main     #509      +/-   ##
==========================================
- Coverage   87.52%   87.37%   -0.16%     
==========================================
  Files          44       44              
  Lines        1988     1996       +8     
  Branches      110      124      +14     
==========================================
+ Hits         1740     1744       +4     
- Misses        248      252       +4
Flag Coverage Δ
unittests 87.37% <63.63%> (-0.16%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...n/scala/com/vertica/spark/config/WriteConfig.scala 100.00% <ø> (ø)
...cala/com/vertica/spark/util/table/TableUtils.scala 80.19% <0.00%> (-1.63%) :arrow_down:
.../vertica/spark/datasource/core/DSConfigSetup.scala 87.74% <66.66%> (-0.47%) :arrow_down:
...e/core/VerticaDistributedFilesystemWritePipe.scala 88.20% <100.00%> (+0.05%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

jeremyprime commented 1 year ago

@ai-bq you'll need to sign the CLA (you should have gotten an email) before that check will pass.

Also note there is a large file at examples/scala/cs that shouldn't be committed.