search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
635 stars 299 forks source link

May need update GlueContext initialisation for spark 3.3 and glue 4.0 #164

Open singlewind opened 1 year ago

singlewind commented 1 year ago

See below warnings for glue 4.0. Just quickly check the code, may be we can pass sparkSession to GlueContext instead of create one.

spark/python/pyspark/sql/context.py:112: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(
dlchet commented 10 months ago

I don't know if this is the best way to handle this, but just to make the warning go away I changed lines 56-57 of context.py to 

    def __init__(self, sparkContext, sparkSession=None, **options):
        if sparkSession is not None:
            super(GlueContext, self).__init__(sparkContext, sparkSession=sparkSession)
        else:
            super(GlueContext, self).__init__(sparkContext)

and it seemed to work if i passed in the session.