Closed kszonsteg closed 3 months ago
The default timeout for jobs in the EMR serverless is 12 hours (https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/considerations.html). For most uses cases this timeout is sufficient, but not for all and especially for the spark structured streaming job, which is running all the time.
The solution proposal is to add the --emr-serverless-timeout flag to the run command and pass the value to the start_job_run method.
Merged
lmouhib
commented
3 months ago lmouhib
commented
3 months ago
The default timeout for jobs in the EMR serverless is 12 hours (https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/considerations.html). For most uses cases this timeout is sufficient, but not for all and especially for the spark structured streaming job, which is running all the time.
The solution proposal is to add the --emr-serverless-timeout flag to the run command and pass the value to the start_job_run method.