vertica / spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Apache License 2.0
20 stars 23 forks source link

Added a complex array and Spark ML Jupyter example #530

Closed ai-bq closed 1 year ago

ai-bq commented 1 year ago

Summary

A common use case for users of the connector is Spark's ML tools. Additionally, Jupyter is an easy way to run examples that allows for the user to experiment and make modifications without having to rebuild examples.

Description

Added an example of Spark's Linear Regression algorithm as well as a complex array writing example.

Related Issue

Closes #436.

Additional Reviewers

@alexey-temnikov @alexr-bq @jonathanl-bq @jeremyp-bq

codecov[bot] commented 1 year ago

Codecov Report

Merging #530 (7d96451) into main (61b2798) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #530   +/-   ##
=======================================
  Coverage   87.55%   87.55%           
=======================================
  Files          44       44           
  Lines        2000     2000           
  Branches      116      114    -2     
=======================================
  Hits         1751     1751           
  Misses        249      249           
Flag Coverage Δ
unittests 87.55% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...la/com/vertica/spark/util/schema/SchemaTools.scala 88.55% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

alexey-temnikov commented 1 year ago

spark-connector/examples/jupyter/README.md should be updated to include this example, as well as describe what this example is about.

alexey-temnikov commented 1 year ago

Command - "docker-compose --profile jupyter up -d" will not work from spark-connector/examples/jupyter/ as there is no docker-compose file. Instructions need to be fixed.

jeremyprime commented 1 year ago

Do we need to add/update the examples here based on this PR?: https://github.com/vertica/vertica-demo/pull/155

ai-bq commented 1 year ago

Do we need to add/update the examples here based on this PR?: vertica/vertica-demo#155

Yes, I will compare the two and update this set.