k8s: async job tracker - Githubissues

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)

Apache License 2.0

26 stars 4 forks source link

k8s: async job tracker #235

Closed jdries closed 1 year ago

jdries commented 2 years ago

Job tracker is a micro service for batch job management. On K8S, job tracker tasks are still part of the main web service. We need to see how this can be deployed separately, and if that requires a nifi instance.

jdries commented 2 years ago

Should be possible to use K8S cron job, but need a solution to register usage in marketplace.

bossie commented 1 year ago

As discussed:

cron job should be a matter of writing a YAML file and specifying the Docker image;
reporting usage to Marketplace is analogous to what we do in Nifi.

bossie commented 1 year ago

Added a job-tracker CronJob to the spark-jobs namespace in Kubernetes. The JobTracker thread that ran in the Flask process has been disabled.

A workaround for the OpenEO web app relying on the JobTracker having downloaded the job results to disk is in place until #232 is implemented; this assumption no longer holds because they run in different containers now.

bossie commented 1 year ago

As discussed: usage reporting, relying on an ETL API, will be tackled in #272.