IPv6 Support for broker and proxy with bindAddress

[bpereto]

Describe the bug the default setting for binding in broker.conf and proxy.conf is bindAddress=0.0.0.0. https://github.com/apache/pulsar/blob/3e065718939d028a970c6b03b96da94e78217583/conf/proxy.conf#L53

the specified default 0.0.0.0 binds to ipv4 only, so the service is not reachable over ipv6.

[root@proxy pulsar]# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:6650            0.0.0.0:*               LISTEN      41186      8351118    2470721/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      41186      8349589    2470721/java

this setting does not accept a a ipv6 address [::0], ::0 or a ipv6 address with or without brackets.

available error in logs:

2022-03-17T10:52:55,986+0100 [sun.misc.Launcher$AppClassLoader@1d56ce6a] error Uncaught exception in thread main: Failed to bind Pulsar Proxy on port 6650

To Reproduce Steps to reproduce the behavior:

1. Set a ipv6 address in broker.conf or proxy.conf for bindAdress: bindAddress=[::0]
2. restart pulsar broker or proxy
3. check listening ports - no pulsar listen on ::6650

Expected behavior pulsar broker and proxy listens on ipv6 address.

Desktop (please complete the following information):

• OS: RedHat 8
• Pulsar v2.9.1

Additional context

• related issue: https://github.com/apache/pulsar/issues/8231
• There was a related merge request for the "advertised service urls", which claims to resolve the issue, but its only resolved for service urls, not the binding/listening: https://github.com/apache/pulsar/issues/954

[eolivelli]

are you able to troubleshot why this is happening ? like putting some debug

[bpereto]

unfortunately, setting log levels to debug with env vars:

PULSAR_LOG_LEVEL=debug
PULSAR_LOG_ROOT_LEVEL=debug

does show debug logs, but no additional errors or hints for this specific problem.

is jetty used for this? the default bind address is set to 0.0.0.0 in the codebase: https://github.com/apache/pulsar/blob/342a5df24ca9dcf142e471ac0aafebc253042ceb/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java#L227

and there is a jetty discussion, if the bind address is not defined, both stacks (4 & 6) are considered by default: https://www.eclipse.org/lists/jetty-users/msg09643.html

[eolivelli]

we are using Jetty for the REST API part and Netty for the Binary part

[bpereto]

error originating from: https://github.com/apache/pulsar/blob/3a337b8952239136cb77302e22021b3d79154f1c/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java#L210

I think this is the relevant part for binding in netty: https://github.com/netty/netty/blob/d34212439068091bcec29a8fad4df82f0a82c638/transport/src/main/java/io/netty/bootstrap/AbstractBootstrap.java#L252

references here for the properties, where lobmok is used:
https://github.com/apache/pulsar/blob/3a337b8952239136cb77302e22021b3d79154f1c/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java#L143

convert with SocketUtils.socketAddress string to InetSocketAddress: https://github.com/netty/netty/blob/40382fb1af275e931a765c5001f4b6bd8061991e/common/src/main/java/io/netty/util/internal/SocketUtils.java#L177

but i don't see the thrown exception somewhere in the output or log, but there should be one from the thrown IOException

[eolivelli]

if I remove all the occurrences of "-Djava.net.preferIPv4Stack=true" inside "bin" (or you can set it to "false") then I am able to boot Pulsar standalone and the proxy on my Mac with JDK11

[bpereto]

jup, this works.

set bindAddress=0.0.0.0 in proxy.conf and removing -Djava.net.preferIPv4Stack=true in bin/pulsar is indeed working.

why is the setting java.net.preferIPv4Stack even in place? Should this not be as the default (false) and enabling dualstack?
IMHO this should be standard today.

[eolivelli]

I believe that it is only because support for IPV6 is not fully available in many libraries (like parsing addresses..) and no one dares to enable it.

If you are able to successfully run all Pulsar components without issues, then please advertise on user@pulsar or dev@pulsar and we can remove that flag, or make it configurable somehow.

[wangjialing218]

I have already test IPV6 connections between pulasr client and broker, as long as kafka client and kop, with -Djava.net.preferIPv4Stack=false, there seems no problem. And I'm going to have a IPV6 test between broker and bookie and zookeeper, after than we could make it default false.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[bpereto]

/remove stale

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[bpereto]

/remove stale

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[MarvinCai]

hi @bpereto, are you expecting any additional changes in Pulsar to support ipv6? As Enrico mentioned, we added -Djava.net.preferIPv4Stack=true as most users are still using ipv4. And to override this behavior without change the ./bin/pulsar script we can add env variable export PULSAR_EXTRA_OPTS=-Djava.net.preferIPv4Stack=false, this should be able to override the the default setting.

[bpereto]

Hi @MarvinCai,

We are currently on 2.9 and did not test the available refactor in master (https://github.com/apache/pulsar/commit/58ad46722c25301a995f798eeec92366ed01ffc6).

An available ENV variable as you suggesting would be nice, also to support adoption.

[bpereto]

tested with

• pulsar 3.0.5 LTS
• jdk 17
• PULSAR_EXTRA_OPTS=-Djava.net.preferIPv4Stack=false
• BOOKIE_EXTRA_OPTS=-Djava.net.preferIPv4Stack=false

and seems to work so far.