Standalone cluster testing process

[Aryex]

Summary

The PR made changes to facilitate testing the connector against a distributed environment.

Changes

Docker environment:

• Added a spark standalone cluster using bitnami's Spark container image. This allows us to easily change the Spark install and worker count.
• Updated docker_client_1 Dockerfile to be based on bitnami's Spark containers to make use of the Spark install.
  • Both docker_client_1 and cluster Spark version are synced
• sandbox-clientenv scripts are updated with options to change the worker count.

Functional testing

• Updated functional tests with option -r to set up test suites for use with spark-submit.
• submit-functional-tests.sh updated to accommodate changes. Also added support to pass in all parameters.
• Workflow updated.

Other

• Nightly workflow execution schedule updated from Sunday - Thursday to Monday - Friday.

Related Issue

Close #413 Close #466

Additional Reviewers

@alexey-temnikov @alexr-bq @jeremyp-bq @jonathanl-bq

[codecov[bot]]

Codecov Report

Merging #465 (7b16183) into main (ecaba43) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #465   +/-   ##
=======================================
  Coverage   87.27%   87.27%           
=======================================
  Files          43       43           
  Lines        1965     1965           
  Branches      118      118           
=======================================
  Hits         1715     1715           
  Misses        250      250           

Flag	Coverage Δ
unittests	87.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more