search

MrPowers / mack

Delta Lake helper methods in PySpark
https://mrpowers.github.io/mack/
MIT License
303 stars 44 forks source link

Unit test over multiple python versions #118

Open ianwesleyarmstrong opened 1 year ago

ianwesleyarmstrong commented 1 year ago

I started with the currently supported python versions and any minor versions of delta lake > 1.0 - I'm happy to adjust these as needed. I thought it also might be worthwhile to test over different versions of pyspark, so I included the minor versions > 3 as well:

pyspark-version: ["3.1", "3.2", "3.3", "3.4"]
ianwesleyarmstrong commented 1 year ago

I've been running the tests on my fork and it appears that python 3.11 may cause some problems - I'll remove that for the time being.

https://github.com/ianwesleyarmstrong/mack/actions/runs/5652132175

MrPowers commented 1 year ago

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I don't think mack supports Delta 1.0.

Here are the version combinations I think we should run (Delta Lake version | PySpark version):

2.4.x | 3.4.x 2.3.x | 3.3.x 2.2.x | 3.3.x 2.1.x | 3.3.x 2.0.x | 3.2.x

This comes from the docs.

ianwesleyarmstrong commented 11 months ago

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I modified it to only test those pairs listed in the docs using Python 3.10 like you suggested

Successful run

Happy to make any other suggestion improvements you find! @MrPowers

andreale28 commented 10 months ago

Hi @MrPowers, Is there any updates on this branch? I'm looking forward to using mack on pyspark 3.4.1 and delta-spark 2.4.0.