AlexIoannides / pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.
1.56k stars 672 forks source link

Leverage chispa to clean up the example test #15

Closed MrPowers closed 3 years ago

MrPowers commented 3 years ago

Thanks for creating this sample project for the community!

I wrote spark-fast-tests to help with Scala testing and recently created a PySpark test helper library called chispa.

chispa provides nicely formatted error messages and actually errored out with this output until I set ignore_nullable = True:

Screen Shot 2020-07-08 at 7 55 16 PM

Thanks again!

AlexIoannides commented 3 years ago

Sorry, but I want to keep this as high-level and dependency-free as is possible.