AWS Glue Workflow failed

[sunbc0120]

step failed after running

glue.wait_for_workflow_finished(config.GLUE_WORKFLOW, glue_run_id)

It says

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-8-350db00730e8> in <module>()
----> 1 glue.wait_for_workflow_finished(config.GLUE_WORKFLOW, glue_run_id)

~/SageMaker/src/package/data/glue.py in wait_for_workflow_finished(workflow_name, workflow_run_id, polling_frequency)
     39     workflow_name, workflow_run_id, polling_frequency=10
     40 ):
---> 41     while not workflow_finished(workflow_name, workflow_run_id):
     42         sleep(polling_frequency)
     43         print('.', end='', flush=True)

~/SageMaker/src/package/data/glue.py in workflow_finished(workflow_name, workflow_run_id)
     29         else:
     30             raise Exception(
---> 31                 "AWS Glue Workflow failed. "
     32                 "Check the workflow logs on the console."
     33             )

Exception: AWS Glue Workflow failed. Check the workflow logs on the console.

In the Glue console, it says

AnalysisException: 'org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.openx.data.jsonserde.JsonSerDe not found);'

[thomelane]

Many thanks for the report @sunbc0120.

Just wondering, did you edit the etl_job.py script in anyway? When you deployed 2 days ago I believe the AWS Glue Job was exporting a CSV file and so it shouldn't be using the JsonSerDe.

Strangely I have just (this second) released an update to the AWS Glue Job to export JSON, so JsonSerDe is now used. Just want to make sure my development workflow isn't spilling into release solutions here.

[sunbc0120]

Thanks, @thomelane . I confirm I'd never touched etl_job.py when this issue happened. But everything seems working now :)

I'll close this issue then. Thanks for this GREAT work!