search

malexer / pytest-spark

pytest plugin to run the tests with support of pyspark
MIT License
85 stars 30 forks source link

findspark.init(spark_home) is never called #4

Closed swhitelaw closed 6 years ago

swhitelaw commented 6 years ago

in pytest_configure, find_spark_home_var is called, which will raise an error if it can't find spark_home. But, find_spark_home_var never calls findspark.init(spark_home), which previously could automatically find the spark_home without being set anywhere by looking in common locations. This means we cannot run any pytests that were previously running due to not being able to fine spark_home

swhitelaw commented 6 years ago

UPDATE: The issue actually is that it raises an error without checking the PYTHONPATH. In our workflow, we specify the spark locations via the PYTHONPATH rather than with SPARK_HOME. This has been a valid way to be able to use pyspark with pytest and the configure should not raise an error if this is the case. The previous version of this code did not raise an error, which was allowed the code to run successfully.

malexer commented 6 years ago

@swhitelaw thanks for the report! I have removed raising the error and this logic should be similar to the one in v 0.4.0 which I believe you were using before.

Please, check the new version 0.4.4 (already on pypi). Let me know if it solves your issue.

swhitelaw commented 6 years ago

Hi @malexer, I believe that fixed our issue. Tested with the following commands without SPARK_HOME defined: pip install pytest-spark --upgrade pytest --version Previously this command would fail with the raised error. It now runs without issue. Thanks for the quick response and fix. I have one more test to confirm. Will update when that test is completed.

malexer commented 6 years ago

Great! BTW you can also override all settings by command line option pytest --spark_home=<path>

swhitelaw commented 6 years ago

UPDATE: confirming that this fix did resolve the issue.