Closed goldengrisha closed 2 years ago
We apologize for delay. Cloud you please share the reproduction step? I was not able to reproduce this issue.
Hello, all is ok, it was resolved. Thank you.
@goldengrisha I am running into this same issue, what did you do to resolve it?
Never mind, I figured it out. It had to do with how I was creating the Glue Context:
from awsglue.context import GlueContext
from pyspark.sql.session import SparkSession
from pyspark.sql.types import (
StructField,
StructType,
StringType,
)
def _add_column(rec):
rec["pk"] = "1"
return rec
def test_serializer_error():
# WRONG WAY! Will throw error
glue_context = GlueContext(SparkSession.builder.getOrCreate())
# CORRECT_WAY! Pass in Spark Context
glue_context = GlueContext(SparkSession.builder.getOrCreate().sparkContext)
dyf = glue_context.create_dynamic_frame.from_rdd(
data=[("test",)],
name="DynamicFrame",
schema=StructType(
[
StructField("test", StringType(), True),
]
),
)
mapped_dyf = dyf.map(f=_add_column)
The documentation for using the docker image found here has examples of creating the Glue Context with the first method. This is confusing since that will cause this error.
Please help, I use
aws-glue-libs:glue_libs_3.0.0_image_01
from docker hub, and all the time I get errors like: