Need Different Resource Profiles to Choose From Per Job

[ranchodeluxe]

Problem

Not all jobs will have the same memory/cpu needs and it's tricky business to weed these out

Solution

While generally over provisioning is needed if we care about saving resources we might need to provide different resource profiles to pick from for a job. Some things to think about

AC:

• multiple node groups should target different AWS instance types
• job/task managers should then have taints/tolerations for each on that can schedule them on the right resource
• resource profiles will have to be an option in the runner and GH Actions inputs

[ranchodeluxe]

this naive version of this has been implemented. now looking at:

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler