search

tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.72k stars 526 forks source link

Cannot connect with IPv6 addresses #1589

Open lantw44 opened 1 year ago

lantw44 commented 1 year ago

Describe the bug Cannot connect to the local Kafka server listeners=PLAINTEXT://[::1]:9092 with [::1]:9092.

To Reproduce

new Kafka({ brokers: [ '[::1]:9092' ] })

Expected behavior It should be able to connect.

Observed behavior It tried to connect to a weird host and failed.

[2023-06-12 10:13:09.421 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:09.424 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 0
    retryTime: 332
[2023-06-12 10:13:10.759 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:10.759 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 1
    retryTime: 594
[2023-06-12 10:13:12.210 +0800] ERROR (ocpi-cloud/15374): Connection error: getaddrinfo ENOTFOUND [
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
    stack: "Error: getaddrinfo ENOTFOUND [\n    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)\n    at __node_internal_ (node:internal/errors:712:10)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)"
[2023-06-12 10:13:12.211 +0800] ERROR (ocpi-cloud/15374): Connection error: getaddrinfo ENOTFOUND [
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
    stack: "Error: getaddrinfo ENOTFOUND [\n    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)\n    at __node_internal_ (node:internal/errors:712:10)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)"
[2023-06-12 10:13:12.356 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:12.358 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 2
    retryTime: 1344
[2023-06-12 10:13:14.705 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:14.706 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 3
    retryTime: 2902
[2023-06-12 10:13:16.206 +0800] ERROR (ocpi-cloud/15374): Connection error: getaddrinfo ENOTFOUND [
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
    stack: "Error: getaddrinfo ENOTFOUND [\n    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)\n    at __node_internal_ (node:internal/errors:712:10)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)"
[2023-06-12 10:13:16.208 +0800] ERROR (ocpi-cloud/15374): Connection error: getaddrinfo ENOTFOUND [
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
    stack: "Error: getaddrinfo ENOTFOUND [\n    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)\n    at __node_internal_ (node:internal/errors:712:10)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)"
[2023-06-12 10:13:18.610 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:18.611 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 4
    retryTime: 5454
[2023-06-12 10:13:21.455 +0800] ERROR (ocpi-cloud/15374): Connection error: getaddrinfo ENOTFOUND [
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
    stack: "Error: getaddrinfo ENOTFOUND [\n    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)\n    at __node_internal_ (node:internal/errors:712:10)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)"
[2023-06-12 10:13:25.067 +0800] ERROR (ocpi-cloud/15374): Connection timeout
    module: "[Redacted]"
    namespace: "Connection"
    broker: "[:0"
    clientId: "kafkajs"
[2023-06-12 10:13:25.069 +0800] ERROR (ocpi-cloud/15374): Failed to connect to seed broker, trying another broker from the list: Connection timeout
    module: "[Redacted]"
    namespace: "BrokerPool"
    retryCount: 5
    retryTime: 9006
[2023-06-12 10:13:25.073 +0800] ERROR (ocpi-cloud/15374): Start service error
    err: {
      "type": "UnknownError",
      "message": "Kafka producer connect error: Connection timeout",
      "stack":
          UnknownError: Kafka producer connect error: Connection timeout
              at <anonymous> ([Redacted])
              at [Redacted]/node_modules/neverthrow/dist/index.cjs.js:113:102
              at <anonymous> ([Redacted])
              at main ([Redacted])
      "error": {
        "type": "KafkaJSNumberOfRetriesExceeded",
        "message": "Connection timeout: Connection timeout",
        "stack":
            KafkaJSNonRetriableError
              Caused by: KafkaJSConnectionError: Connection timeout
                at Timeout.onTimeout [as _onTimeout] ([Redacted]/node_modules/kafkajs/src/network/connection.js:223:23)
                at listOnTimeout (node:internal/timers:569:17)
                at process.processTimers (node:internal/timers:512:7)
            caused by: KafkaJSConnectionError: Connection timeout
                at Timeout.onTimeout [as _onTimeout] ([Redacted]/node_modules/kafkajs/src/network/connection.js:223:23)
                at listOnTimeout (node:internal/timers:569:17)
                at process.processTimers (node:internal/timers:512:7)
        "name": "KafkaJSNumberOfRetriesExceeded",
        "retriable": false,
        "retryCount": 5,
        "retryTime": 9006
      },
      "name": "UnknownError"
    }

Environment:

OneCricketeer commented 1 year ago

Do your brokers really need ipv6 addresses? You could use 

export KAFKA_OPTS="-Djava.net.preferIPv4Stack=true"
kafka-server-start ...

And use an IPv4 listener/advertised listener

lantw44 commented 1 year ago

Do your brokers really need ipv6 addresses?

No, but it is surprising to know that a popular library still can't properly support IPv6 addresses in 2023.

ban-shi-yi-sheng commented 6 months ago

就是就是 ipv6都不支持

Emile83 commented 1 month ago

Hi there, we also faced the same issue with kafkajs not supporting ipv6 addresses, with the difference that the configuration is made by our customers and if they go for ipv6 we cannot bypass and use ipv4.

Fortunately we overcame this limitation by overwriting the connectionPoolBuilder.js at build stage. It's all about properly parsing the broker list in case of ipv6 (assuming kind of [ipv6]:port). That worked like a charm after that, meaning the rest of the connection logic totally supports ipv6.

In my opinion it is still not the best way to do, since it remains tricky and heavy to parse ipv4, ipv6 with or without port and with or without brackets. The code may get much simpler if instead of a list of brokers as string kafkajs would accept a list of broker objects having two fields: {"host": "string", "port?": number}. However it would break compatibilities.

Regards