[Tech Debt] Update configuration on Databricks tests

Is this a new bug in dbt-spark?

[X] I believe this is a new bug in dbt-spark
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

These tests are failing for the Databricks test profiles:

tests/functional/adapter/test_python_model.py::TestPySpark::test_different_dataframes
tests/functional/adapter/test_python_model.py::TestChangingSchemaSpark::test_changing_schema_with_log_validation

Expected Behavior

All tests should pass

Steps To Reproduce

Appears in all CI

Relevant log output

https://github.com/dbt-labs/dbt-spark/actions/runs/9617443575/job/26531074677?pr=1053

Environment

- OS: any
- Python: any
- dbt-core: any
- dbt-spark: any

Additional Context

The test failures appear to be our configuration of the test environment and do not represent broken functionality in dbt-spark itself. This break also appears related to the release of numpy==2.0 and the fact that we're indirectly forcing the use of Python 3.7 in these tests.

dbt-labs / dbt-spark

[Tech Debt] Update configuration on Databricks tests #1054