search

GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
407 stars 119 forks source link

Refine test sequencing and versions #1303

Closed nj1973 closed 2 weeks ago

nj1973 commented 1 month ago

Integration test sequence Currently Teradata, Snowflake, DB2 and Hive integration tests run sequentially after all other tests have completed. This is particularly painful because the Hive tests are the longest running tests.

We should try changing cloudbuild.yaml to include waitFor: ['-'] so that they all run concurrently. Hopefully the higher concurrency doesn't cause any infrastructure issues. If it does then perhaps we can rearrange things slightly so Hive at least starts as soon as possible.

MySQL wantFor incorrect MySQL uses the proxy to connect and should have the following waitFor line:

waitFor: ['proxy-install']

PYTHON_VERSIONS We should review PYTHON_VERSIONS = ["3.8", "3.9", "3.10"] in noxfile.py.

I think we should add 3.11 and 3.12 to this list.

helensilva14 commented 2 weeks ago

Cloud Build updates worked with no problems having all steps running concurrently!

But unfortunately, adding 3.11 and 3.12 to the Python versions list will need to be addressed in a separate issue because first we need to make sure all clients work properly for each one of the versions. So far, I was able to get errors from Oracle and Hive with 3.12. And since we randomly choose a Python version for each system, it's not something that we can have certainty right now on when it will be selected or not.

nj1973 commented 2 weeks ago

Python versions list will need to be addressed in a separate issue

I agree, it is not because of client compatibility though. I believe our python-multi image doesn't have more recent versions of Python:

Step #10 - "integration_oracle": nox > Session integration_oracle failed: Python interpreter 3.12 not found.
helensilva14 commented 2 weeks ago

That makes sense! I created #1321 as a follow-up and I'll finish the PR with just the Cloud Build changes.