nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.92k stars 1.41k forks source link

Issues with auth using system account #2651

Closed lesovsky closed 3 years ago

lesovsky commented 3 years ago

I have a test nats cluster with 3 nodes, with configured authentication. The config of first node looks like

port: 4222
http: 8222
syslog: true
pid_file: /var/lib/nats-server/nats.pid

server_name: nats1
jetstream: true

authorization {
  user1 = {
    publish = ">"
    subscribe = ">"
  }

  users = [
    {user: admin, password: "qwerty"}
    {user: user1, password: "qwerty", permissions: $user1 }
  ]
}

accounts: {
    SYS: {
        users: [
            { user: admin, password: qwerty }
        ]
    },
}

system_account: SYS

cluster {
  listen: 0.0.0.0:5222
  name: test-cluster

  authorization {
      user: admin
      password: qwerty
  }

  routes: [
     "nats-route://admin:qwerty@nats2:5222"
     "nats-route://admin:qwerty@nats3:5222"
  ]
}

jetstream: {
    store_dir: /var/lib/nats-server
    max_memory_store: 1GB
    max_file_store: 1GB
}

Configs for the rest nodes are similar except server_name and cluster.routes.

After starting services, everything looks fine and no errors in the log.

But when I try to check cluster status using direct connection to specific nodes with nats server report jetstream --server nats://admin:qwerty@nodeX:4222 command I got correct answer only from single host. Other hosts respond with error nats: error: server request failed, ensure the account used has system privileges and appropriate permissions, try --help. I expected correct answer from the all hosts.

First thing I thought was wrong creds in the config. But all configs are deployed using Ansible, I checked deployed configs and found no mistakes in creds. Next I tried to restart nats-servers and found that second server started respond correctly. Made several restarts of third service I achieved that all servers started respond correctly :) I am sure this is a wrong behavior and have to be fixed.

I also started nats-servers with -DVV flags and catched the following output from valid and invalid nodes: valid:

Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - Client connection created
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"admin","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.26","lang":"go","version":"1.12.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [PING]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [PONG]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [SUB _INBOX.JVXjMK14zkvDoWHbcIM9P3  1]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [HPUB $SYS.REQ.SERVER.PING.JSZ _INBOX.JVXjMK14zkvDoWHbcIM9P3 37 39]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- MSG_PAYLOAD: ["NATS/1.0\r\nAccept-Encoding: snappy\r\n\r\n{}"]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [HMSG _INBOX.JVXjMK14zkvDoWHbcIM9P3 1  38 544]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [HMSG _INBOX.JVXjMK14zkvDoWHbcIM9P3 1 38 564]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [HMSG _INBOX.JVXjMK14zkvDoWHbcIM9P3 1 38 563]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [UNSUB 1 ]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - <-> [DELSUB 1]
Oct 28 09:46:52 nats3 nats-server[21420]: 192.168.122.1:38832 - cid:12 - "v1.12.0:go:NATS CLI Version 0.0.26" - Client connection closed: Client Closed

invalid:

Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - Client connection created
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"admin","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.26","lang":"go","version":"1.12.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [PING]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [PONG]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [SUB _INBOX.3QNrkFFxbnlwQltIw8B3b3  1]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [HPUB $SYS.REQ.SERVER.PING.JSZ _INBOX.3QNrkFFxbnlwQltIw8B3b3 37 39]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- MSG_PAYLOAD: ["NATS/1.0\r\nAccept-Encoding: snappy\r\n\r\n{}"]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [UNSUB 1 ]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - <-> [DELSUB 1]
Oct 28 09:44:08 nats1 nats-server[21550]: 192.168.122.1:58704 - cid:11 - "v1.12.0:go:NATS CLI Version 0.0.26" - Client connection closed: Client Closed

I also created a natscli-context where specified all nodes and when I make connections to nats using the context I got random errors. image

mannharleen commented 3 years ago

Am facing similar issues with nats@latest. I am using the docker image nats

derekcollison commented 3 years ago

You have two users that are the same but bind to different accounts. {user: admin, password: "qwerty"} is present for the implicit $G global account and the SYS account.

Try this..

port: 4222
http: 8222
syslog: true
pid_file: /var/lib/nats-server/nats.pid

server_name: nats1
jetstream: true

authorization {
  user1 = {
    publish = ">"
    subscribe = ">"
  }

  users = [
    {user: user_admin, password: "qwerty"}
    {user: user1, password: "qwerty", permissions: $user1 }
  ]
}

$SYS { users = [ { user: "admin", pass: "qwerty" } ] }

cluster {
  listen: 0.0.0.0:5222
  name: test-cluster

  authorization {
      user: admin
      password: qwerty
  }

  routes: [
     "nats-route://admin:qwerty@nats2:5222"
     "nats-route://admin:qwerty@nats3:5222"
  ]
}

jetstream: {
    store_dir: /var/lib/nats-server
    max_memory_store: 1GB
    max_file_store: 1GB
}
lesovsky commented 3 years ago

You have two users that are the same but bind to different accounts. {user: admin, password: "qwerty"} is present for the implicit $G global account and the SYS account.

Ok, I see the point. Removed cluster admin user from users list and everything works fine now. Thanks.

rodrigc commented 2 years ago

@jnmoyne clarified the docs for configuring a system account in https://github.com/nats-io/nats.docs/pull/493 See: https://docs.nats.io/running-a-nats-service/configuration/clustering/jetstream_clustering#configuration

Sagarbud commented 5 months ago

Hello all and @derekcollison

I have installed NATS on GKE cluster using Helm. How i can enable SYS account and Global account. Also, How we can configure permission for users.

derekcollison commented 5 months ago

In a nats.conf file I just add these to the bottom to add in a $SYS user when using the default global account.

accounts { $SYS: { users = [ {user: "admin", password: "s3cr3t!"} ] } }

Would need @wallyqs or @caleblloyd to weigh in on how you do that with a helm chart.

Sagarbud commented 5 months ago

Thank you @derekcollison for quick response!

Hello @wallyqs and @caleblloyd

I have installed NATS on GKE cluster using Helm. How i can enable SYS account and Global account. Also, How we can configure permission for users.

Sagarbud commented 5 months ago

Hello @wallyqs and @caleblloyd

I have added below auth section in values.yaml file.

auth: enabled: true basic: users:

nats context add dev-nats --server 10.0.0.1 --description "test dev" --user admin --password pass --select

nats server info

Can you please help me to resolve the issue?