vertica / spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Apache License 2.0
20 stars 23 forks source link

[BUG] S3/GCS tests use same staging directory for each test #539

Open ai-bq opened 1 year ago

ai-bq commented 1 year ago

Recently upon attempting to run the Weekly tests for both Java 11 and 8 it was discovered that the S3/GCS tests for 11 and 8 cannot run at the same time due to the fact that the staging directory is the same throughout each test.

It was since decided that Java 8 only needed it's own end to end test and all though the GCS/S3 tests are functional and passing, the end to ends should have unique staging directories in case of issues in the future.

One of the failed weekly tests can be seen here.

In particular:

Array("***/94d9f456_dd4c_45b8_860c_4957e902b697") was not empty. After each test, staging directory should be cleaned.