search

dbt-labs / spark-utils

Utility functions for dbt projects running on Spark
https://hub.getdbt.com/fishtown-analytics/spark_utils/latest/
Apache License 2.0
31 stars 15 forks source link

Add testing #22

Closed JCZuurmond closed 2 years ago

JCZuurmond commented 2 years ago

This PR is a proposal for adding testing using pytest. I would like to try this first here. If we are happy with the set-up, I want to move this into a python package so that it can be used for all dbt-spark projects.

JCZuurmond commented 2 years ago

As just said in https://github.com/dbt-labs/dbt-spark/issues/272, I will have a first go at a package for unit testing dbt-spark logic at the end of the month.

I would like some guidance on where to integrate with dbt-core. Let's discuss those in the comments above.

jtcohen6 commented 2 years ago

@JCZuurmond This is really neat!! Your work has sparked and anticipated many conversations internally, as we're working to refactor our own testing framework.

If you're interested, check out https://github.com/dbt-labs/dbt-core/pull/4691. We're just exiting the exploratory phase, and experimenting toward a set of concrete proposals. I'd love to hear your feedback over there, and (if you're up for it) find ways to include you in our ongoing work.

JCZuurmond commented 2 years ago

@jtcohen6 : Curious to get feedback again! Do you think this way of testing benefits the project? Will you merge this PR?

JCZuurmond commented 2 years ago

Thanks for reviewing the PR, @jtcohen6. I have not tested the library against other adapters than Spark. The lib is written adapter agnostic, but I expect it is easiest to use on in-memory databases.

I am keen to include others on the project or to move parts of it to another project. If you could help with that, that would be great. I have ideas about what I want to change further, however, I prefer to include others first. To hear what others think and to allow the library to grow to be community based.

JCZuurmond commented 2 years ago

@jtcohen6 : Could you have another look?

JCZuurmond commented 2 years ago

The CI fails due to tests/functional/test_utils.py expecting a dbt.tests.adapter module.