AWS cost attribution: split each hub in all of AWS clusters into their own user nodegroups

[GeorgianaElena]

For each cluster in the list below, perform the following actions:

- [ ] Create the same set of nodes for each hub of the cluster
- [ ] Configure each of the hubs so their users only spawn on the nodepools designated for them
- [ ] Tag each of the nodepool with `2i2c:hub-name` to match the name of the hub

• [ ] 2i2c-aws-us
• [ ] catalystproject-africa
• [ ] gridsst
• [ ] jupyter-health
• [ ] kitware
• [ ] nasa-cryo
• [ ] opensci
• [ ] smithsonian
• [ ] strudel
• [ ] ubc-eoas
• [x] victor
• [ ] nmfs-openscapes
• [x] openscapes
• [x] earthscope
• [ ] jupyter-meets-the-earth
• [x] nasa-ghg
• [x] nasa-veda
• [x] projectpythia (they are using other cost allocation system https://github.com/2i2c-org/meta/issues/1612)

Reference

See https://github.com/2i2c-org/infrastructure/issues/4482 for reference as at the end, their config should be similar to one of openscapes.

Definition of done

• [ ] Users across the hubs of all clusters, will spawn on to their own nodepools, and not share them with users on other hubs.
• [ ] There is a tag 2i2c:hub-name and 2i2c:node-purpose on all the nodes spawned when users log on to the hub. You can verify this by looking at EC2 instances on the AWS console.
• [ ] the cost of each hub shows up in its respective grafana

[sgibson91]

@GeorgianaElena do you want to split up the list of clusters between us? E.g. I'm currently working on nmfs-openscapes so may as well do that.

[GeorgianaElena]

@sgibson91, I'll start open separate issues for each cluster so that we can split them more easily: self-assign -> do the work -> close the issue. And to get that nice feeling of fixing an issue rather than have a single big one.

I will start with the nmfs-openscapes one so you can self-assign and start working on it if you want 🚀

[sgibson91]

Sounds great!

[GeorgianaElena]

@sgibson91, let's use this comment to record with what clusters we ran into issues so we can come back to them later on. So far we had issues with:

• nmfs-openscapes -> both of us had issues running eksctl commands against this
• jmte -> tracked by #5101