search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
669 stars 131 forks source link

SPARK_VERSION not found #146

Closed rajeshraj124 closed 11 months ago

rajeshraj124 commented 11 months ago

Dear Team,

I am encountering a challenge while configuring jobs in AWS Glue. I am currently using Glue version 4.0 along with PyDeequ 1.1.0 and either of the jars 'deequ-2.0.2-spark-3.3' or 'deequ-2.0.2-spark-3.3'. Could anyone kindly suggest how to address this issue?

In AWS Glue, we're unsure about how to create the 'SPARK_VERSION' environment variable since we haven't found any clear guidance on this.

image

### Tasks
rajeshraj124 commented 11 months ago

refer to this ticket #138