numaproj / numaflow-python

Numaflow Python SDK
Apache License 2.0
50 stars 17 forks source link

Update multiproc to use UDS instead of TCP #161

Closed kohlisid closed 4 months ago

kohlisid commented 4 months ago

Summary

As now we use multiple servers instead of one in multiproc, we can transition them to use UDS sockets instead of TCP. This is also required for container updates as if we use TCP on any server restart the ports are refreshed and the numa container isnt able to reconnect. To solve this we can 1) Fix the TCP port list 2) Transition to fixed UDS sockets

Number 2 seems more plausible as individually UDS has better performance than TCP


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.