Running pytest with local spark session

[rotemb-cye]

Hey,

I am trying to run pytest on my local PC, when databricks extension is installed. I am trying to create local spark session:

def get_spark_session():
    spark = (
        SparkSession.builder.master("local[*]")
        .appName("local-tests")
        .config("spark.driver.bindAddress", "127.0.0.1")
        .getOrCreate()
    )
    return spark

@pytest.mark.etl
@pytest.fixture(scope="session")
def spark_session():
    spark = get_spark_session()
    yield spark
    spark.stop()

and I get the following error: RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

How to solve it? I want to be able to run my pytest when being offline

TNXXX

[htuomola]

Hi, this is the same exact issue that we have been struggling with. It seems that installing databricks-connect modifies installed pyspark package and adds throwing this error to the code. I'm also interested in finding a workaround for this because in the current state it basically blocks using Databricks Connect.

[benoitLebreton-perso]

Hello I managed to get my local spark session working by the following VSCode command palette

In fact, even uninstalling the extension was not working.

[odimko]

thanks for your solution @benoitLebreton-perso! Do you know how to fix the issue when you run pytest from a command line?