KubeMQ binds only on IPv6 interfaces

[rweland]

My Setup: 1 Node k3s deployment on IPv4 ONLY lab network Other workloads up and working as expected KubeMQ Community latest release (as of 2/25/22) deployed using default config

What happens: When the community edition is deployed it starts, however all listeners only bind to the containers IPv6 address. This makes it impossible to connect to the container with the CLI from an IPv4 network.

What's expected: Listeners should bind to both IPv4 and IPv6 interfaces.

Digging in the code, the raw TCP socket listeners appear to be correctly setting the bind port according to the net.Listen documentation. As well the echo servers appear to be setup correctly as well. Ref: grpc/server.go:111 lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port)) Ref: rest/server.go:172 _ = s.echoWebServer.Start(fmt.Sprintf(":%d", conf.Port))

However as seen from the startup log output, it appears to ignore this.

2022-02-26T02:07:21.322Z    INFO    starting kubemq community edition version: , cores: 16  {"host": "f1a8683371dc", "module": "server"}
2022-02-26T02:07:21.323Z    INFO    ⇨ http server started on [::]:8080
    {"host": "f1a8683371dc", "module": "server-api"}
2022-02-26T02:07:22.324Z    INFO    Starting broker-server version 2.1.9    {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z    INFO    Git commit [not set]    {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z    INFO    Listening for client connections on 0.0.0.0:39983   {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z    INFO    Server id is NAT3D4HT53N6MRBB4T35ZVRXEPEC3CSRAWKTVZ5JXGCI3TWT2772BX2E   {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z    INFO    Server is ready {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.325Z    INFO    In-Memory client server connection pipe: ya3PsP8PSdGJarJJX9lIIC started {"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.350Z    INFO    STREAM: Starting broker-streaming-server[kubemq] version 0.19.0 {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z    INFO    STREAM: ServerID: 08rN2ijHYGp37AYGf7AYqI    {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z    INFO    STREAM: Go version: go1.17.7    {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z    INFO    STREAM: Git commit: [not set]   {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.352Z    INFO    STREAM: Recovering the state... {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.352Z    INFO    STREAM: No recovered state  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: Message store is FILE   {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: Store location: store/f1a8683371dc  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: ---------- Store Limits ----------  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: Channels:            unlimited  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: --------- Channels Limits --------  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM:   Subscriptions:     unlimited  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM:   Messages     :     unlimited  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM:   Bytes        :     unlimited  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM:   Age          :       24h0m0s  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM:   Inactivity   :       24h0m0s  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: ----------------------------------  {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z    INFO    STREAM: Streaming Server is ready   {"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:23.603Z    INFO    routing service loaded  {"host": "f1a8683371dc", "module": "services"}
2022-02-26T02:07:23.606Z    WARN    started insecure grpc server at port 50000  {"host": "f1a8683371dc", "module": "grpc"}
2022-02-26T02:07:23.606Z    WARN    grpc interface is accepting traffic {"host": "f1a8683371dc", "module": "grpc"}
2022-02-26T02:07:23.606Z    WARN    rest interface is accepting traffic {"host": "f1a8683371dc", "module": "rest-interface"}
2022-02-26T02:07:23.606Z    INFO    started insecure rest server at port 9090   {"host": "f1a8683371dc", "module": "rest-interface"}
2022-02-26T02:07:23.606Z    INFO    ⇨ http server started on [::]:9090
    {"host": "f1a8683371dc", "module": "rest-api"}
2022-02-26T02:07:23.606Z    INFO    STREAM: Channel "_QUEUE_DELAY_" has been created    {"host": "f1a8683371dc", "module": "broker-store-server"}

This is confirmed by remoting into the container and listing the bound ipv4 and ipv6 ports Open IPv4 ports

I have no name!@f1a8683371dc:/kubemq$ declare -a array=($(tail /proc/net/tcp | cut -d":" -f"3"|cut -d" " -f"1")) && for port in ${array[@]}; do echo $((0x$port)); done | sort | uniq     
57560

Open IPv6 ports

I have no name!@f1a8683371dc:/kubemq$ declare -a array=($(tail /proc/net/tcp6 | cut -d":" -f"3"|cut -d" " -f"1")) && for port in ${array[@]}; do echo $((0x$port)); done | sort | uniq    
36742
39983
42349
50000
8080
9090

Things I've tried:

• Rebuild container from source
• Rebuild with different base image (bullseye-slim)
• Hardcoded address/ports (ex 0.0.0.0:50000 for grpc)
• Hardcoded ListenerNetwork for echo servers to tcp

The only thing that has successfully allowed the project to bind on ipv4 was to set the first param of Listen and the ListenerNetwork property of the echo servers to tcp4, which of course forced IPv4 only.

2022-02-26T03:00:41.577Z INFO ⇨ http server started on 0.0.0.0:9090

I've run out of things to try to allow both ipv4 and ipv6, any ideas?

[rweland]

I found this post in the golang issues https://github.com/golang/go/issues/9334 which seems to be a long standing issue with the net package of golang. The issue doesn't appear to be going anywhere at the moment.

I found a similar issue in echo servers history https://github.com/labstack/echo/issues/1658. Their solution was to add the aforementioned configuration setting I referenced in the issue. I think that may be the solution to go with.

I have put in a PR #21 to add transport as a config setting for api, grpc, and rest if that is the direction we want to take.

[kubemq]

PR Merged