search

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
https://celeborn.apache.org/
Apache License 2.0
862 stars 351 forks source link

[CELEBORN-1513] Support wildcard bind in dual stack environments #2713

Closed akpatnam25 closed 1 week ago

akpatnam25 commented 2 weeks ago

What changes were proposed in this pull request?

Support wildcard bind for RPC and HTTP servers. When wildcard address is used, the service is able to listen to both ipv4 and ipv6 traffic in dual-stack environments.

The specific scenario where this becomes relevant is as follows:

If some of the compute infrastructure is IPv4 only, some v6 only and others dual stack - the way we can have a single Celeborn infra to cater to all is by: a) Set bind.preferip to false - so that advertised address is the host and not IP.

b) bind to wild card address

With both in place, the v4 only compute nodes will resolve the v4 address and connect to v4 ip/port. Likewise, for v6 only. Dual stack compute nodes will use prefer ipv6 Java flag to resolve to either v4 or v6.

This is how we are handling the combination of scenarios internally.

Why are the changes needed?

see above.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tested on a server using netstat, and tried connecting to via nc -4 and nc -6 to ensure connection was there.

akpatnam25 commented 2 weeks ago

this is a duplicate of https://github.com/apache/celeborn/pull/2633 -- I had forgotten to update it for a long time and it became stale with a lot of conflicts, hence I just made a new branch/PR.

akpatnam25 commented 2 weeks ago

cc @FMX @SteNicholas @pan3793 @RexXiong @turboFei please review, thank you :)

akpatnam25 commented 1 week ago

thanks @RexXiong , I have addressed your comments. Please take another look , thanks!! :)

akpatnam25 commented 1 week ago

@RexXiong @SteNicholas are we able to merge this? :)

akpatnam25 commented 1 week ago

thanks everyone for the reviews!!