vertica / spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Apache License 2.0
20 stars 23 forks source link

Add a Jupyter Notebook container #508

Closed jeremyprime closed 1 year ago

jeremyprime commented 1 year ago

Summary

Add a Jupyter Notebook container

Description

Added a Jupyter Notebook container, under a profile so that it is not started each time by default. And also added a single Jupyter Notebook example showing how to read and write from Vertica using the Spark Connector. We can simply add more examples to the examples/jupyter/ folder in the future.

Note that we do need to keep the Spark and Python versions in-sync between the Jupyter and Spark containers, otherwise Jupyter will not work. So for now we are changing the Python version in the Spark container to match the one in Jupyter.

The main, daily, and weekly tests all pass with these changes (i.e. the change to use a modified version of the Bitnami image with a different version of Python does not affect the tests, it only affects the ability to use a particular version of Jupyter Notebook as the Spark and Python versions must match).

Related Issue

Closes #488.

Additional Reviewers

@alexey-temnikov @alexr-bq @ai-bq @jonathanl-bq

codecov[bot] commented 1 year ago

Codecov Report

Merging #508 (c8436b3) into main (00d0c05) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #508   +/-   ##
=======================================
  Coverage   87.52%   87.52%           
=======================================
  Files          44       44           
  Lines        1988     1988           
  Branches      110      110           
=======================================
  Hits         1740     1740           
  Misses        248      248           
Flag Coverage Δ
unittests 87.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more