node pool tech debt

[kevindrosendahl]

known issues:

• [ ] service waits for node pool to be stable before updating the deployment. this is the right case when a node pool needs a new epoch or is scaling up, but when scaling down the pods are not terminated gracefully
• [ ] every node update triggers a node pool sync, which results in at least a terraform plan, and node updates occur every 15s for each node
• [ ] currently using labels for path and name, which impose a 63 char limit. probably want to switch to using a label to indicate that it is a system level node pool, and an annotation to indicate the full path of the system node pool
• [ ] service controller complicated by node pool possibly being nil when using a shared node pool
• [ ] label selector introduces 9 character limit on system name:

  E0625 19:03:49.119322       1 node_pool_controller.go:209] lattice-system-e2e-system-823926/d70b831e-6763-4dd2-8a20-8b5396abb2fe failed with : error getting status for node pool d70b831e-6763-4dd2-8a20-8b5396abb2fe (dedicated for service 52ace3d2-08e5-4a70-a9c3-faf6b6a1674b in system e2e-system-823926) epoch 1: error making requirement for node pool d70b831e-6763-4dd2-8a20-8b5396abb2fe (dedicated for service 52ace3d2-08e5-4a70-a9c3-faf6b6a1674b in system e2e-system-823926) node lookup: invalid label value: "lattice-system-e2e-system-823926.d70b831e-6763-4dd2-8a20-8b5396abb2fe.1": must be no more than 63 characters

[kevindrosendahl]

should split label value (currently concatenated .. into three separate labels (and taints). this will solve the name character limit