Currently pipelines and downloads are using Stackable Spark CRD templates for submitting Spark/Beam jobs using dynamic settings for executors, cores, memory and CPU, this should be tested and probably modified to assure those settings are understood by Yunikorn, an example of that config was committed for downloads in the Airflow DAGs repository
Currently pipelines and downloads are using Stackable Spark CRD templates for submitting Spark/Beam jobs using dynamic settings for executors, cores, memory and CPU, this should be tested and probably modified to assure those settings are understood by Yunikorn, an example of that config was committed for downloads in the Airflow DAGs repository