search

eakmanrq / sqlframe

Turning PySpark Into a Universal DataFrame API
https://sqlframe.readthedocs.io/en/stable/
MIT License
174 stars 3 forks source link

feat: add support for pyspark testing #86

Closed eakmanrq closed 2 weeks ago

eakmanrq commented 2 weeks ago

Added assertDataFrameEqual and assertSchemaEqual support for all engines. Since this diffing requires access to the schema this is not available when using Standalone.

I haven't personally used this part of the PySpark API so I don't have much intuition in how it would be used. Looking for user feedback on where to invest time into next for this.

Resolves: https://github.com/eakmanrq/sqlframe/issues/28

This also includes schema/printSchema support for Spark since it was an oversight in past PRs that added this support.