search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

Can MQTT work in Super-cluster with Gateways? #2252

Closed wade19870531 closed 3 years ago

wade19870531 commented 3 years ago

I want to use mqtt with nats super cluster. Two nats clusters and a super cluster were created with gateways with following config.

pid_file: "/var/run/nats/nats.pid"
http: 8222
server_name: $POD_NAME
cluster {
  name: nats-server-1
  port: 6222
  routes: [nats://myuser:mypass@nats-server-1:6222]
  authorization: {
    user: myuser
    password: mypass
    timeout: 0.75
  }
  cluster_advertise: $CLUSTER_ADVERTISE
  connect_retries: 60
}
operator: /KO/KO.jwt
resolver: URL(http://nats-accounts-1:9090/jwt/v1/accounts/)
system_account: ACSLANO6BCHBYBGYGITGLNF6EAVDZ7HCIMF7XW5CRDO7SFSGDBBQDOP2
gateway: {
    authorization: {
        user: myuser
        password: mypass
    }
    name: nats-server-1
    port: 7222
    advertise: "nats-server-1:7222"
    gateways: [{"name":"nats-server-1","url":"nats://myuser:mypass@nats-server-1:7222"},{"name":"nats-server-2","url":"nats://myuser:mypass@nats-server-2.myns:7222"}]
}
leafnodes: {
  listen: "0.0.0.0:7422"
}
jetstream: enabled
jetstream {
    store_dir: $DIR
}
mqtt: {
    host: 0.0.0.0
    port: 1883
}

However, when mqtt client connect to nats server, nats server response not authorised

# mosquitto_sub -h nats-server-1 -p 1883 -t test/topic
Connection error: Connection Refused: not authorised.

If nats config remove these section(operator, resolver, system_account, gateway), mqtt client can connect to NATS Server and work.

mqtt sender:

# mosquitto_pub -h nats-server-1 -p 1883 -t test/topic -m "test"

mqtt subscriber:

# mosquitto_sub -h nats-server-1 -p 1883 -t test/topic
test

I also try to connect by mqtt client with nats user, but still not authorised. Do I use a wrong method or any misunderstand? Thanks.

# mosquitto_sub -h mqtt://nats-server-1 -p 1883 -t test/topic --key "SU...QQ" --cert "ey..._hxBw"
Unable to connect (Lookup error.).

error reading protocol name: unexpected EOF(nats-server log)

Versions of nats-server and affected client libraries used:

nats-server: 2.2.5 nats-account-server: 1.0.0 nsc: 2.2.3 natscli: 0.0.23

OS/Container environment:

kubernets 18.5 istio 1.8.0

NATS Server Log:

[68] 2021/05/26 05:47:34.244315 [INF] Starting nats-server
[68] 2021/05/26 05:47:34.244396 [INF]   Version:  2.2.5
[68] 2021/05/26 05:47:34.244400 [INF]   Git:      [b7e1f66]
[68] 2021/05/26 05:47:34.244403 [INF]   Name:     nats-server-1-0
[68] 2021/05/26 05:47:34.244411 [INF]   Node:     ALzTzbKU
[68] 2021/05/26 05:47:34.244421 [INF]   ID:       NCXGVZEFDEHRIQQPBWWLKGT3QXOKWO6OOQR5J5SJN4OKHNLKA7OBX5AC
[68] 2021/05/26 05:47:34.244430 [INF] Using configuration file: /etc/nats-config/nats.conf
[68] 2021/05/26 05:47:34.244435 [INF] Trusted Operators
[68] 2021/05/26 05:47:34.244439 [INF]   System  : ""
[68] 2021/05/26 05:47:34.244443 [INF]   Operator: "KO"
[68] 2021/05/26 05:47:34.244475 [INF]   Issued  : 2021-05-26 03:06:01 +0000 UTC
[68] 2021/05/26 05:47:34.244483 [INF]   Expires : 1970-01-01 00:00:00 +0000 UTC
[68] 2021/05/26 05:47:34.245730 [INF] Starting JetStream
[68] 2021/05/26 05:47:34.247107 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[68] 2021/05/26 05:47:34.247125 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[68] 2021/05/26 05:47:34.247148 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[68] 2021/05/26 05:47:34.247154 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[68] 2021/05/26 05:47:34.247158 [INF]
[68] 2021/05/26 05:47:34.247163 [INF]          https://docs.nats.io/jetstream
[68] 2021/05/26 05:47:34.247167 [INF]
[68] 2021/05/26 05:47:34.247171 [INF] ---------------- JETSTREAM ----------------
[68] 2021/05/26 05:47:34.247183 [INF]   Max Memory:      17.65 GB
[68] 2021/05/26 05:47:34.247190 [INF]   Max Storage:     37.97 GB
[68] 2021/05/26 05:47:34.247196 [INF]   Store Directory: "/data/jetstream/nats-server-1-0/store/jetstream"
[68] 2021/05/26 05:47:34.247215 [INF] -------------------------------------------
[68] 2021/05/26 05:47:34.247819 [WRN] Account fetch failed: will only fetch valid account keys
[68] 2021/05/26 05:47:34.247882 [INF] Starting JetStream cluster
[68] 2021/05/26 05:47:34.247896 [INF] Creating JetStream metadata controller
[68] 2021/05/26 05:47:34.248577 [INF] JetStream cluster recovering state
[68] 2021/05/26 05:47:34.250521 [INF] Starting http monitor on 0.0.0.0:8222
[68] 2021/05/26 05:47:34.251407 [INF] Gateway name is nats-server-1
[68] 2021/05/26 05:47:34.251436 [INF] Listening for gateways connections on 0.0.0.0:7222
[68] 2021/05/26 05:47:34.251474 [INF] Advertise address for gateway "nats-server-1" is set to nats-server-1:7222
[68] 2021/05/26 05:47:34.251921 [INF] Listening for leafnode connections on 0.0.0.0:7422
[68] 2021/05/26 05:47:34.252417 [INF] Listening for MQTT clients on mqtt://0.0.0.0:1883
[68] 2021/05/26 05:47:34.252479 [INF] Listening for client connections on 0.0.0.0:4222
[68] 2021/05/26 05:47:34.252617 [INF] Server is ready
[68] 2021/05/26 05:47:34.252723 [INF] Cluster name is nats-server-1
[68] 2021/05/26 05:47:34.252758 [INF] Listening for route connections on 0.0.0.0:6222
[68] 2021/05/26 05:47:34.255500 [INF] 10.104.52.249:6222 - rid:6 - Route connection created
[68] 2021/05/26 05:47:34.255801 [INF] 10.104.52.249:6222 - rid:6 - Router connection closed: Client Closed
[68] 2021/05/26 05:47:35.254249 [INF] Connecting to explicit gateway "nats-server-2" (nats-server-2.myns:7222) at 10.108.245.245:7222 (attempt 1)
[68] 2021/05/26 05:47:35.254578 [INF] 10.108.245.245:7222 - gid:7 - Creating outbound gateway connection to "nats-server-2"
[68] 2021/05/26 05:47:35.266059 [INF] 10.104.52.249:6222 - rid:8 - Route connection created
[68] 2021/05/26 05:47:35.266422 [INF] 10.104.52.249:6222 - rid:8 - Router connection closed: Client Closed
[68] 2021/05/26 05:47:35.497093 [INF] 10.108.245.245:7222 - gid:7 - Outbound gateway connection to "nats-server-2" (NARLL4Y6R5OFFAVLRKAEP6C2SZHL62PYAVAXXOXXIICGN6QYNEFYG6MT) registered
[68] 2021/05/26 05:47:36.347023 [INF] 10.104.52.249:6222 - rid:9 - Route connection created
[68] 2021/05/26 05:47:36.347119 [INF] 10.104.52.249:6222 - rid:9 - Router connection closed: Client Closed
[68] 2021/05/26 05:47:37.405350 [INF] 10.104.52.249:6222 - rid:10 - Route connection created
[68] 2021/05/26 05:47:37.405456 [INF] 10.104.52.249:6222 - rid:10 - Router connection closed: Client Closed
[68] 2021/05/26 05:47:37.796636 [INF] 127.0.0.1:43210 - gid:11 - Processing inbound gateway connection
[68] 2021/05/26 05:47:38.073963 [INF] 127.0.0.1:43214 - gid:12 - Processing inbound gateway connection
[68] 2021/05/26 05:47:38.184558 [INF] 127.0.0.1:43234 - gid:13 - Processing inbound gateway connection
[68] 2021/05/26 05:47:38.239805 [INF] 127.0.0.1:43210 - gid:11 - Inbound gateway connection from "nats-server-2" (NA72IDGOY2EF326Q5DKBSI6EY5JZA5DJFDMZGXG537O33NEXHOKYK57B) registered
[68] 2021/05/26 05:47:38.442126 [INF] 10.104.52.249:6222 - rid:16 - Route connection created
[68] 2021/05/26 05:47:38.443129 [INF] 127.0.0.1:57260 - rid:17 - Route connection created
[68] 2021/05/26 05:47:38.453341 [INF] 10.104.52.249:6222 - rid:16 - Router connection closed: Duplicate Route
[68] 2021/05/26 05:47:38.453606 [INF] 127.0.0.1:57260 - rid:17 - Router connection closed: Duplicate Route
[68] 2021/05/26 05:47:38.523139 [INF] 127.0.0.1:43214 - gid:12 - Inbound gateway connection from "nats-server-2" (NARLL4Y6R5OFFAVLRKAEP6C2SZHL62PYAVAXXOXXIICGN6QYNEFYG6MT) registered
[68] 2021/05/26 05:47:38.626237 [INF] 127.0.0.1:43234 - gid:13 - Inbound gateway connection from "nats-server-2" (NA2LK35HFPLY6IXFYNA5GZNSARDACPJGFWPWIR4657J2KVWZXTQTKU3G) registered
[68] 2021/05/26 05:47:40.127999 [INF] 127.0.0.1:59390 - lid:20 - Leafnode connection created
[68] 2021/05/26 05:47:40.128250 [INF] 127.0.0.1:59388 - lid:21 - Leafnode connection created
[68] 2021/05/26 05:47:40.128456 [INF] 127.0.0.1:59392 - lid:22 - Leafnode connection created
[68] 2021/05/26 05:47:45.239611 [INF] 127.0.0.1:57440 - rid:23 - Route connection created
[68] 2021/05/26 05:47:51.378845 [INF] 127.0.0.1:57594 - rid:24 - Route connection created
[68] 2021/05/26 05:48:32.844092 [ERR] 127.0.0.1:39088 - mid:25 - authentication error
[68] 2021/05/26 05:48:32.844190 [ERR] 127.0.0.1:39088 - mid:25 - unable to connect: authentication error
kozlovic commented 3 years ago

@wade19870531 This is expected. In operator mode, all connections are required to present a JWT in order to connect. For MQTT clients, it means providing it through the MQTT password (and use any username you want since it is required to have a username if there is a password). As of now, though, the JWT must have the bearer boolean set to true.

I have a PR for the documentation so the above can be found in the future: https://github.com/nats-io/nats.docs/pull/264

I am also proposing that we remove the bearer requirement for MQTT clients: https://github.com/nats-io/nats-server/pull/2253, but not sure if this will be accepted, and regardless, this would not be available until a future release, so for now you will have to provide a JWT with bearer boolean set to true.

kozlovic commented 3 years ago

Just a note that #2253 is rejected, so you will need to have a JWT with bearer token.

wade19870531 commented 3 years ago

@kozlovic Thanks.