search

malexer / pytest-spark

pytest plugin to run the tests with support of pyspark
MIT License
85 stars 30 forks source link

catch ImportError instead of reading __version__ number #2

Closed clembou closed 7 years ago

clembou commented 7 years ago

Hi again @malexer ... Thanks for releasing a new version of pytest-spark so quickly!

However I found a problem in the version check code I added in the spark_session fixture. pyspark.__version__ is only defined from v2.1 onwards... 😞

This causes an ugly exception instead of showing an explanation message, and most importantly even prevents the fixture from working on spark 2.0 where it should be available.

the spark version can be read from sparkContext.version in older versions but that requires an instances of sparkContext to be created which is not the case here.

Instead I went for a simpler check to fix the issue: catch the ImportError in old spark versions and display a more helpful message.

Sorry about that!

malexer commented 7 years ago

Oh, my fault as well - I have not checked it on the older versions of Spark. Thanks!