As a [developer] I want to evaluate autoscaling in the etl because improve performance and resource usage
Background
The Platform ETL runs using 4 highmem-64 machines and some of the steps use less than 20% of the resources whereas other steps like literature could use more worker nodes to improve performance.
Tasks
[x] Update workflow configuration to include autoscale
[ ] Verify the ETL runs correctly using the autoscale feature
[ ] Compare both configurations to evaluate if the autoscale does improve resource usage and performance.
Acceptance tests
How do we know the task is complete?
The functionality of the ETL is not affected by the new configuration
When applying the autoscale configuration we get better performance and resource utilisation.
As a [developer] I want to evaluate autoscaling in the etl because improve performance and resource usage
Background
The Platform ETL runs using 4 highmem-64 machines and some of the steps use less than 20% of the resources whereas other steps like literature could use more worker nodes to improve performance.
Tasks
Acceptance tests
How do we know the task is complete?