Updating Automated Tests

[Aryex]

Summary

This PR contains updates to the automated testing. In general, each PR now only runs unit tests, scalastyles, and functional test against latest Spark. Regression testing and other tests are done nightly.

Description

Replacing main.yml with on-pull-request.yml workflow, now run unit tests, scalastyle checks, and functional tests against the latest version of Spark.

Updated nightly.yml workflow to perform long-running tests. This includes:

• Regression testing against different Spark 3.x vs Vertica 10, 11, and 12
• Testing against GCS and S3
• Testing JSON option against Spark 3.x vs Vertica 12. This involves updating a few functional tests to handle json option.

Functional tests updated:

• Added an exclude option, allowing the user to exclude test suites from a run.
• Modified checks for a few tests in ComplexTypeTests and EndToEndTests to account for when using JSON option.
• build.sbt no longer specify org.apache.hadoop.hadoop-hdfs dependency. spark-core already defines it.
• build.sbt will now additionally look for sparkVersion and hadoopAwsVersion from the environment variables. Previously it was only checking in JVM's properties.

Readmes updated

• A Readme added to .github/workflows
• Updated S3 to reflect new learning about setting up aws-hadoop connector
• Updated functional tests readme to reflect changes.

Bug fixes:

• Fixed a bug where if the exported JSON folder was empty, the connector would fail as the underlying Spark JSON reader reports nothing to read. Changes start at VerticaJsonScan.batch where after the data was exported and Spark's Json reader has been initialized, we then delete the folder if empty.

Related Issue

Close #421 Close #402 Close #463

Additional Reviewers

[codecov[bot]]

Codecov Report

Merging #457 (532c2e9) into main (070fc7d) will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main     #457   +/-   ##
=======================================
  Coverage   87.27%   87.27%           
=======================================
  Files          43       43           
  Lines        1965     1965           
  Branches      116      118    +2     
=======================================
  Hits         1715     1715           
  Misses        250      250           

Flag	Coverage Δ
unittests	87.27% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ce/core/VerticaDistributedFilesystemReadPipe.scala	89.75% <ø> (ø)
...ertica/spark/datasource/json/VerticaJsonScan.scala	91.66% <ø> (ø)
.../spark/datasource/v2/VerticaDatasourceV2Read.scala	86.04% <100.00%> (ø)
...rtica/spark/util/version/VerticaVersionUtils.scala	97.82% <100.00%> (ø)

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more