search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

teraslice k8s - convert execution controller IP to k8s service #3696

Open godber opened 1 month ago

godber commented 1 month ago

Right now, when a teraslice job starts, we wait for the execution controller pod to come up and get an IP address. We then supply that IP to the workers to communicate with the execution controller. Instead, we could use a Kubernetes service will will provide a stable domain name for the workers to connect to. This would make implementing restartable execution controllers possible.

godber commented 1 month ago

I feel like there is a gotcha here ... like I tried to implement this and failed. So this might not work out for some reason, or perhaps it's just not as simple as it seems at first blush.