Fix Flaky CI Failure: use isolated spark session and randomized namespace

[pengyu-hou]

Summary

The unit test suite is running in parallel in CI. Due to the massive table creations, it might lead to racing condition during different test cases. This PR will create a different randomized namespace for each test case to isolate the table content. It should fix the flaky CI checks.

To further isolate the testing cases, different spark session to avoid interferences among different testing cases.

Why / Goal

Improve code quality and fix flaky testing CI

Test Plan

• [x] Added Unit Tests
• [x] Covered by existing CI
• [ ] Integration tested

Checklist

• [ ] Documentation update

Reviewers

@yuli-han @hzding621