search

malexer / pytest-spark

pytest plugin to run the tests with support of pyspark
MIT License
84 stars 30 forks source link

Python module cannot find spark package #23

Open alpenkar opened 2 years ago

alpenkar commented 2 years ago

I'm attempting to use this library to run pytests on sparkML code using the SynapseML package from microsoft. My code runs fine when using spark-submit, however after including the following in my pytest.ini, I still receive a ModuleNotFound error when trying to run my pytests. relevant code is below:

pytest.ini

[pytest]
spark_options =
    spark.app.name: fw-active-ml-core-pytest-spark-tests
    spark.executor.instances: 1
    spark.jars.packages: com.microsoft.azure:synapseml_2.12:0.9.5

failing code

from synapse.ml.lightgbm import LightGBMRegressor as sparkLightGBMRegressor

error

ImportError while importing test module '/Users/alpenkar/Documents/Code/fw-active-ml-core/tests/unit/test_pipeline_library_flow_operations.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/unit/test_pipeline_library_flow_operations.py:12: in <module>
    from fw_active_ml_core.pipeline import PipelineLibrary as pl
fw_active_ml_core/pipeline/PipelineLibrary.py:14: in <module>
    from fw_active_ml_core.pipeline.PipelineLibraryMLOperations import *
fw_active_ml_core/pipeline/PipelineLibraryMLOperations.py:49: in <module>
    from synapse.ml.lightgbm import LightGBMRegressor as sparkLightGBMRegressor
E   ModuleNotFoundError: No module named 'synapse'

Thanks.