spark_session fixture raises ValueError during setup

[skyegecko]

I get the following stacktrace from pytest when trying to call the spark_session fixture:

ERROR at setup of <testname> 

    @pytest.fixture(scope='session')
    def _spark_session():
        """Internal fixture for SparkSession instance.

        Yields SparkSession instance if it is supported by the pyspark
        version, otherwise yields None.

        Required to correctly initialize `spark_context` fixture after
        `spark_session` fixture.

        ..note::
            It is not possible to create SparkSession from the existing
            SparkContext.
        """

        try:
            from pyspark.sql import SparkSession
        except ImportError:
            yield
        else:
            session = SparkSession.builder \
>               .config(conf=SparkConfigBuilder().get()) \
                .enableHiveSupport() \
                .getOrCreate()

../../../.local/share/virtualenvs/Transform-3Dsmlban/lib/python3.4/site-packages/pytest_spark/fixtures.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'pytest_spark.config.SparkConfigBuilder'>

    @classmethod
    def get(cls):
        if not cls._instance:
>           raise ValueError
E           ValueError

../../../.local/share/virtualenvs/Transform-3Dsmlban/lib/python3.4/site-packages/pytest_spark/config.py:49: ValueError

Taking a quick look at the code, it seems to me that the raise ValueError line is completely unnecessary, since the configobject can just be initialized then?

[skyegecko]

A little bit more searching:

It seems this is caused when there are no spark_options defined in pytest.ini. In pytest_spark/__init__.py the SparkConfigBuilder().initialize() function is only called if options are defined, otherwise initialize() is skipped and leads to the later crash.

pytest_spark/fixtures.py is still a bit strange since there is an attempt to call initialize() after the raised ValueError, which will never work.

[forbaokhanh]

• 1 on this issue, running into the same problem. Seems to have started happening 14 days ago when a config file refactor happened

[skyegecko]

I'm currently working on a PR for this.

[malexer]

@dutchgecko thanks for the report and PR and sorry for the late reply

my fault: line raise ValueError was added last minute to check some edge cases and I forgot to remove it

[malexer]

fixed in 0.5.2. Please find it on pypi