Storing the TargetCluster field only in the TaskDefinition level and not persisting it down to the Task level. The abstraction should only be exposed in the DB level.
TargetCluster will take precedence over other ways of defining a cluster name. This enables definition-level flexibility in defining clusters.
This was not set up for EMR jobs because having virtual clusters already allows us to specify EMR jobs to run in another cluster. There's also some coupling in the actual EKS cluster itself to manage virtual clusters so it can't be entirely handled on the code level. This means that multi-cluster support for EMR is more complicated. We can split the traffic from EMR and other jobs, but not in EMR itself.
New enforcement that a cluster should be defined in the environment variables for it to be considered a valid cluster to run on. This means that an override cluster must be already defined in the main list of clusters to run with.
First pass. Not tested yet.
Notes:
TargetClusterfield only in theTaskDefinitionlevel and not persisting it down to theTasklevel. The abstraction should only be exposed in the DB level.TargetClusterwill take precedence over other ways of defining a cluster name. This enables definition-level flexibility in defining clusters.