This PR adds a new environment variable, GOOGLE_DATAPROC_BATCHES_TTL, to allow us to control Dataproc Batches TTL setting. In newer Batches versions this defaults to 4 hours and when breached cancels the job with a successful status which is not what we want, in v1.1 the parameter defaults to "run forever".
This PR also adds a check after the gcloud call to abort the offload if the batch state is CANCELLED or FAILED.
I've defaulted the GOOGLE_DATAPROC_BATCHES_TTL variable to 2 days in offload.env. I'm open to debate whether that is a good number or not.
This PR adds a new environment variable, GOOGLE_DATAPROC_BATCHES_TTL, to allow us to control Dataproc Batches TTL setting. In newer Batches versions this defaults to 4 hours and when breached cancels the job with a successful status which is not what we want, in v1.1 the parameter defaults to "run forever".
This PR also adds a check after the
gcloudcall to abort the offload if the batch state is CANCELLED or FAILED.I've defaulted the GOOGLE_DATAPROC_BATCHES_TTL variable to 2 days in
offload.env. I'm open to debate whether that is a good number or not.