Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

job_tracker: killing of long running batch jobs #737

Open jdries opened 3 months ago

jdries commented 3 months ago

It's not uncommon to see batch jobs gettings stuck for multiple days, piling up costs. It should not be too hard for job tracker to notice this, and perhaps also to compute intermediate credit usage. This would allow auto-killing of jobs if user credits are exceeded, or simply after a certain amount of time.

Maybe the auto-kill time threshold could be a job option, to still allow running jobs for very long if required? That would allow setting a default of say 1 hour, so that users are force to set something sensible themselves rather fast?