Right now, when a teraslice job starts, we wait for the execution controller pod to come up and get an IP address. We then supply that IP to the workers to communicate with the execution controller. Instead, we could use a Kubernetes service will will provide a stable domain name for the workers to connect to. This would make implementing restartable execution controllers possible.
I feel like there is a gotcha here ... like I tried to implement this and failed. So this might not work out for some reason, or perhaps it's just not as simple as it seems at first blush.
Right now, when a teraslice job starts, we wait for the execution controller pod to come up and get an IP address. We then supply that IP to the workers to communicate with the execution controller. Instead, we could use a Kubernetes service will will provide a stable domain name for the workers to connect to. This would make implementing restartable execution controllers possible.