Update the spark version to the current version - Githubissues

dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks

https://getdbt.com

Apache License 2.0

395 stars 221 forks source link

Update the spark version to the current version #1055

Closed mikealfare closed 3 months ago

mikealfare commented 3 months ago

resolves #1054

Problem

we're using an old runtime version of Spark for some tests
there's an issue in spacy that requires pydanic>=1.10.8 (see https://github.com/explosion/spaCy/issues/12659)
there's an issue in databricks.koalas that requires an update in handling numpy updates; however, this library is no longer supported

Solution

update the version in the tests
update pydanic<1.10.3 to pydanic>=1.10.8
add a docstring explaining all of this
exclude the koalas test

Checklist

[x] I have read the contributing guide and understand what's expected of me
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

github-actions[bot] commented 3 months ago

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.