search

gbif / stackable

GBIF Stackable Infrastructure
Apache License 2.0
4 stars 0 forks source link

Investigate how to submit Jobs to Yunikorn using dynamic settings #21

Closed fmendezh closed 4 months ago

fmendezh commented 8 months ago

Currently pipelines and downloads are using Stackable Spark CRD templates for submitting Spark/Beam jobs using dynamic settings for executors, cores, memory and CPU, this should be tested and probably modified to assure those settings are understood by Yunikorn, an example of that config was committed for downloads in the Airflow DAGs repository

zaultooz commented 4 months ago

Dynamic configuration is enabled by providing these properties when submitting the SparkApplication:

https://github.com/gbif/gbif-airflow-dags/blob/develop/dags/templates/spark_job_template_spark3_5_0.yaml#L44-L49