Unable to Create GlueContext via GlueContext Function in Local Python/awsglue Environment

[adamfortuno]

I'm having the issue described in issue #42.

I am attempting to run the following in my local PySpark console...

from awsglue.context import GlueContext
glueContext = GlueContext(sc)

We receive the following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\XYZ\bin\aws-glue-libs\PyGlue.zip\awsglue\context.py", line 47, in __init__
  File "C:\Users\XYZ\bin\aws-glue-libs\PyGlue.zip\awsglue\context.py", line 68, in _get_glue_scala_context
TypeError: 'JavaPackage' object is not callable

The following is the complete picture:

The environment looks like the following:

• OS: 10.0.17134.0
• Python: 3.7.3
• Hadoop (winutils.exe): 2.8.5
• Spark: 2.4.3
• PySpark: 2.4.6
• awsglue: 1.0

My environment variables look like the following...

• SPARK_HOME: \bin\spark-2.4.3-bin-hadoop2.8\
• SPARK_CONF_DIR: \bin\aws-glue-libs\conf\
• HADOOP_HOME: \bin\hadoop-2.8.5\
• SPARK_CONF_DIR: \bin\spark-2.4.3-bin-hadoop2.8\
• JAVA_HOME: C:\Progra~2\Java\jdk1.8.0\
• CLASSPATH:
  • \bin\aws-glue-libs\jarsv1*
  • \bin\spark-2.4.3-bin-hadoop2.8\jars*
• PYTHONPATH:
  • ${SPARK_HOME}\python\lib\py4j
  • \bin\aws-glue-libs\PyGlue.zip

Just to confirm which version awsglue repo I'm working with...

The following are the "netty" files in my ..\aws-glue-libs\jarsv1\:

I'm looking for a little guidance on how to tweak my configuration to resolve this issue.

[calleo]

Typically this is due to an misconfigured classpath. Check if you have the AWS jars on the classpath.

[adamfortuno]

The driver and executor settings were missing from my spark-defaults.conf file. I made the following additions:

spark.driver.extraClassPath         C://Users/XYZ/bin/aws-glue-libs/jarsv1/*
spark.executor.extraClassPath       C://Users/XYZ/bin/aws-glue-libs/jarsv1/*

Everything worked as expected.