Open royraposonjr opened 1 year ago
Well, it was a bit tricky, I tried including the jars for core and storage but it didn't register the classes for some reason. I ended up including it in the packages which did work for me.
here is an example of my pytest fixture that initiates the spark session for the tests
from awsglue.context import GlueContext
from pyspark.sql import SparkSession
import pytest
@pytest.fixture(scope="session", autouse=True)
def glue_context():
spark_session = (
SparkSession
.builder
.config("spark.jars.packages", "io.delta:delta-core_2.12:2.1.0")
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
.getOrCreate()
)
glue_context = GlueContext(spark_session.sparkContext)
yield glue_context
spark_session.stop()
I have a problem with using native delta table support even after adding DATALAKE_FORMATS: delta to my environment which adds the jars for delta-core. It still cant import delta module.
A workaround I found is to add --py-files /home/glue_user/aws-glue-libs/datalake-connectors/delta-2.1.0/delta-core_2.12-2.1.0.jar to my arguments and on pytest I added spark.sparkContext.addPyFile("/home/glue_user/aws-glue-libs/datalake-connectors/delta-2.1.0/delta-core_2.12-2.1.0.jar").
Is there a way to automatically load the jars or am I missing something?