Create a simple Airflow DAG that trigger an occurrence downloads and tracks it status.
The DAG must trigger a Spark job that and its settings must be applied dynamically to assign cores, executors and memory accordingly to the amount of data to process.
Once the DAGs finishes its status must be updated in the registry database.
[ ] Implement a simple DAG that triggers an download Spark job.
[ ] Implement a basic mechanism to assign resources to spark jobs based on the amount of records to process by a download.
Create a simple Airflow DAG that trigger an occurrence downloads and tracks it status. The DAG must trigger a Spark job that and its settings must be applied dynamically to assign cores, executors and memory accordingly to the amount of data to process. Once the DAGs finishes its status must be updated in the registry database.