Improve infrastructure around Hyperscale tests

[hanefi]

We have 2 seperate test schedules for hyperscale: multi_schedule_hyperscale that contains the tests that can be run as unprivileged user and multi_schedule_hyperscale_superuser that contains tests that need a privileged user.

Here are some ideas that can improve our test infrastructure for hyperscale schedules:

• [ ] Do not turn off MX by default. https://github.com/citusdata/citus/blob/2cdd49be5d43cfe4e48e479889bc20d79a977f90/src/test/regress/multi_schedule_hyperscale#L22
• [ ] Run the tests on Marlin instead of Meru
• [ ] Marlin has "$user, public" as the default schema hence citus schema comes before public. This causes a lot of changes in the test output.
• [ ] Revisit the contents of the 2 schedules and add more test files to them. Developers were expected to add new test files in that schedule, but that did not happen in the recent past.
• [ ] Raise awareness in the team about a policy on how to maintain these test infrastructure.
• [ ] Some test names are changed and we don't do the necessary changes on these schedules.
• [ ] Some of the normalization rules are not generic enough to capture things when hostname is not localhost, so this also causes some differences.
• [ ] Server side COPY commands that access test data files do not work for running tests on remote clusters. Consider changing them into client-side psql \copy commands as we did in https://github.com/citusdata/citus/pull/6138

Related:

• https://github.com/citusdata/citus/pull/5042
• https://github.com/citusdata/citus/issues/6138#issuecomment-1208690447

[onderkalaci]

Open question: Can we point arbitrary configs to HSC cluster such that we do not maintain multiple test suites.

Currently HSC schedule might have more coverage, but we are gradually making arbitrary configs more powerful

[hanefi]

Open question: Can we point arbitrary configs to HSC cluster such that we do not maintain multiple test suites.

It may be possible after we solve some blockers/deal breakers:

1. We need privileged user in most of our tests. Getting access for that is easy on development clusters but it slightly defeats the purposes of running tests on HyperScale if we do not run the tests using a role with limited privileges.
2. citus_add_node and similar commands may not work in HyperScale.
3. Connecting to some other worker nodes might not work on HyperScale. Though I am not so sure if we have such tests.

Currently HSC schedule might have more coverage, but we are gradually making arbitrary configs more powerful

I agree on both statements here. I also would like to see arbitrary configs run on a HyperScale cluster in the future.

[onderkalaci]

We need privileged user in most of our tests. Getting access for that is easy on development clusters but it slightly defeats the purposes of running tests on HyperScale if we do not run the tests using a role with limited privileges.

arbitrary config runs with a regular user, so shouldn't be a problem. Even if there is 1 schedule that runs superuser -- which I think there is -- we can easily skip that on on HSC.

citus_add_node and similar commands may not work in HyperScale.

Arbitrary config tests do not use those, as they run as non-superuser. There is an initialization step which does that, and that part should not run on HSC.

Connecting to some other worker nodes might not work on HyperScale. Though I am not so sure if we have such tests.

On Citus 11+, you can enable worker query access, even from portal.