search

AlexIoannides / pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.
1.56k stars 672 forks source link

etl_config.json not loaded in EMR #21

Open junjchen opened 3 years ago

junjchen commented 3 years ago

First of all, thanks for the great work! I am new to spark and this repo has really helped me getting started.

I am trying to get my etl job running on aws EMR in cluster mode, but got hit with an issue where the pyspark program failed to load up my config.json passed in through --files s3://path.... I googled a bit and figured this might be an issue with how --files only copies files to executor nodes, not driver node, so the code here https://github.com/AlexIoannides/pyspark-example-project/blob/master/dependencies/spark.py#L93 finds nothing when iterating through the spark root dir. (for reference, I based my guess on this SO post https://stackoverflow.com/questions/47187533/files-option-in-pyspark-not-working).

Could this be the issue and is there a workaround for it? Or am I not making sense at all?

moongyu123 commented 2 years ago

In my case I maked config_json.py in folder 'dependencies' It returns JSONString

and spark.py's code like this from dependencies import config config_json_string = config.get_config_json() config_dict = json.loads(config_json_string)