search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.56k stars 1.39k forks source link

how to enabled JetStream for account #2367

Closed rickiey closed 6 months ago

rickiey commented 3 years ago

$ nats --user=root --password=aaaaaa account info

Connection Information:

               Client ID: 17
               Client IP: 127.0.0.1
                     RTT: 152.236µs
       Headers Supported: true
         Maximum Payload: 1.0 MiB
       Connected Cluster: nc
           Connected URL: nats://127.0.0.1:4222
       Connected Address: 127.0.0.1:4222
     Connected Server ID: NDCVXNEA6YCSXRQEBE62XKMBDPMADOANPQH45TBWF26GVFBCT2XTO5DU
   Connected Server Name: nats-server-150

JetStream Account Information:

   Could not obtain account information: JetStream not enabled for account

listen: "0.0.0.0:4222"

jetstream { store_dir=/var/lib/nats/storage max_mem: 30Gb max_file: 60Gb }

accounts: { USERS: { jetstream: enable users: [ {user: root, password: aaaaaa} ] }, SYS: { users: [ {user: admin, password: aaaaaa} ] }, } system_account: SYS

cluster {

host/port for inbound route connections from other server

name: nc listen: "192.168.55.150:4244"

Authorization for route connections

Other server can connect if they supply the credentials listed here

This server will connect to discovered routes using this user

authorization { user: root password: aaaaaa timeout: "0.5" }

connect_retries: 5

This server establishes routes with these server.

This server solicits new routes and Routes are actively solicited and connected to from this server.

Other servers can connect to us if they supply the correct credentials

in their routes definitions from above.

routes: [ nats-route://root:aaaaaa@192.168.55.151:4244 nats-route://root:aaaaaa@192.168.55.152:4244 ] }

rickiey commented 3 years ago

I didn't see anything about this. Can I give a completed JetStream Cluster configuration document? Your document is really chaotic.

ripienaar commented 3 years ago

Your configuration is correct and enables JetStream, maybe something was logged on the server to indicate a problem at startup?

lorenzodonini commented 3 years ago

I've been experiencing something similar. I have the following configuration:

debug = true
trace = true
server_name = nats.myserver

port = 4222
monitor_port = 8222

jetstream = {
  store_dir = "/data/nats-server/"
  max_memory_store = 52428800
  max_file_store = 5368709120
}

# Authorization
authorization = {
  timeout = 3
  ADMIN = {
    publish = ">"
    subscribe = ">"
  }
  USER = {
    publish = "SANDBOX.*"
    subscribe = ["PUBLIC.>", "_INBOX.>", "someTopic1.user.*."]
  }
  users = [
    {user: $ADMIN_USER,  password: $ADMIN_USER, permissions: $ADMIN}
    {user: 'backend', password: 'backend', permissions: $ADMIN}
    {user: 'myservice', password: 'myservice', permissions: $ADMIN}
    {user: 'user', password: 'user', permissions: $USER}
  ]
}

# Accounts and stream config
accounts = {
  # Regular application users, no jetstream access
  USER = {
    users = [
      {user: 'user', password: 'user'}
    ]
    imports = [
      {stream: {subject: someTopic1.user.*, account: NOTIFICATION}}
    ]
  }
  # Service
  NOTIFICATION = {
    users = [
      {user: 'myservice', password: 'myservice'}
      {user: 'backend', password: 'backend'}
    ]
    jetstream = {
      max_mem: 24M
      max_file: 1G
      max_streams: 5
      max_consumers: 5
    }
    exports = [
      {stream: someTopic1.>}
    ]
  }
  ADMIN = {
    users = [
      {user: $ADMIN_USER, password: $ADMIN_USER}
    ]
    imports = [
      {stream: {subject: someTopic1.>, account: NOTIFICATION}}
    ]
    jetstream = enabled
  }
  # System
  SYS = {
    users = [
      {user: 'sys', password: $SYS}
    ]
  }
}
no_auth_user = user
system_account = SYS

websocket = {
    port = 4223
    no_tls = true
}

I start nats in one container, then connect to it from a second container and get this error:

[1] 2021/08/30 13:21:04.791624 [INF] Starting nats-server
[1] 2021/08/30 13:21:04.791653 [INF]   Version:  2.4.0
[1] 2021/08/30 13:21:04.791656 [INF]   Git:      [e49eb66]
[1] 2021/08/30 13:21:04.791658 [DBG]   Go build: go1.16.7
[1] 2021/08/30 13:21:04.791659 [INF]   Name:     nats.myserver
[1] 2021/08/30 13:21:04.791663 [INF]   Node:     QQNpzpjc
[1] 2021/08/30 13:21:04.791665 [INF]   ID:       NBLT6EMTRPWV72GDWVA2A3RBEJMPDETFLP7WWTSWDVTPRBIFEKE4MUPS
[1] 2021/08/30 13:21:04.791668 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[1] 2021/08/30 13:21:04.791670 [INF] Using configuration file: /etc/nats/nats-server.conf
[1] 2021/08/30 13:21:04.792107 [INF] Starting JetStream
[1] 2021/08/30 13:21:04.792226 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[1] 2021/08/30 13:21:04.792230 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[1] 2021/08/30 13:21:04.792232 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[1] 2021/08/30 13:21:04.792234 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[1] 2021/08/30 13:21:04.792236 [INF] 
[1] 2021/08/30 13:21:04.792238 [INF]          https://docs.nats.io/jetstream
[1] 2021/08/30 13:21:04.792240 [INF] 
[1] 2021/08/30 13:21:04.792242 [INF] ---------------- JETSTREAM ----------------
[1] 2021/08/30 13:21:04.792249 [INF]   Max Memory:      50.00 MB
[1] 2021/08/30 13:21:04.792252 [INF]   Max Storage:     5.00 GB
[1] 2021/08/30 13:21:04.792254 [INF]   Store Directory: "/data/nats-server/jetstream"
[1] 2021/08/30 13:21:04.792256 [INF] -------------------------------------------
[1] 2021/08/30 13:21:04.792341 [DBG]   Exports:
[1] 2021/08/30 13:21:04.792345 [DBG]      $JS.API.>
[1] 2021/08/30 13:21:04.792370 [DBG] Enabled JetStream for account "ADMIN"
[1] 2021/08/30 13:21:04.792375 [DBG]   Max Memory:      -1 B
[1] 2021/08/30 13:21:04.792377 [DBG]   Max Storage:     -1 B
[1] 2021/08/30 13:21:04.792498 [DBG] JetStream state for account "ADMIN" recovered
[1] 2021/08/30 13:21:04.792519 [DBG] Enabled JetStream for account "NOTIFICATION"
[1] 2021/08/30 13:21:04.792524 [DBG]   Max Memory:      22.89 MB
[1] 2021/08/30 13:21:04.792528 [DBG]   Max Storage:     953.67 MB
[1] 2021/08/30 13:21:04.792650 [DBG] JetStream state for account "NOTIFICATION" recovered
[1] 2021/08/30 13:21:04.793673 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2021/08/30 13:21:04.794040 [INF] Listening for websocket clients on ws://0.0.0.0:4223
[1] 2021/08/30 13:21:04.794045 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
[1] 2021/08/30 13:21:04.794050 [DBG] Get non local IPs for "0.0.0.0"
[1] 2021/08/30 13:21:04.883136 [DBG]   ip=192.168.240.2
[1] 2021/08/30 13:21:04.883196 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2021/08/30 13:21:04.883205 [DBG] Get non local IPs for "0.0.0.0"
[1] 2021/08/30 13:21:04.907550 [DBG]   ip=192.168.240.2
[1] 2021/08/30 13:21:04.907564 [INF] Server is ready
[1] 2021/08/30 13:21:08.206993 [DBG] 192.168.240.4:40935 - cid:7 - Client connection created
[1] 2021/08/30 13:21:08.207326 [DBG] 192.168.240.4:40935 - cid:7 - Client connection closed: Client Closed
[1] 2021/08/30 13:21:08.343782 [DBG] 192.168.240.4:54286 - cid:8 - Client connection created
[1] 2021/08/30 13:21:08.344134 [TRC] 192.168.240.4:54286 - cid:8 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"backend","pass":"[REDACTED]","tls_required":false,"name":"backend","lang":"go","version":"1.11.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[1] 2021/08/30 13:21:08.344252 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [PING]
[1] 2021/08/30 13:21:08.344258 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - ->> [PONG]
[1] 2021/08/30 13:21:08.344411 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [SUB _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.*  1]
[1] 2021/08/30 13:21:08.344430 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- [PUB $JS.API.INFO _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.8E6jNhjp 0]
[1] 2021/08/30 13:21:08.344435 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <<- MSG_PAYLOAD: [""]
[1] 2021/08/30 13:21:08.344790 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - ->> [MSG _INBOX.RCZ5h4JPVT4KS5k8r9zKuB.8E6jNhjp 1 145]
[1] 2021/08/30 13:21:08.344814 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  579]
[1] 2021/08/30 13:21:08.344836 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"xeFl4pV2T0KFl5y0yE2w34\",\"timestamp\":\"2021-08-30T13:21:08.344674154Z\",\"server\":\"nats.myserver\",\"client\":{\"start\":\"2021-08-30T13:21:08.343739133Z\",\"host\":\"192.168.240.4\",\"id\":8,\"acc\":\"$G\",\"user\":\"backend\",\"name\":\"backend\",\"lang\":\"go\",\"ver\":\"1.11.0\",\"rtt\":403272,\"server\":\"nats.myserver\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"error\\\":{\\\"code\\\":503,\\\"err_code\\\":10039,\\\"description\\\":\\\"JetStream not enabled for account\\\"}}\"}"]
[1] 2021/08/30 13:21:08.348156 [DBG] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - Client connection closed: Client Closed
[1] 2021/08/30 13:21:08.348179 [TRC] 192.168.240.4:54286 - cid:8 - "v1.11.0:go:backend" - <-> [DELSUB 1]
[1] 2021/08/30 13:21:11.469922 [DBG] 192.168.240.4:54306 - cid:10 - Client connection created
[1] 2021/08/30 13:21:11.470305 [TRC] 192.168.240.4:54306 - cid:10 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"backend","pass":"[REDACTED]","tls_required":false,"name":"backend","lang":"go","version":"1.11.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[1] 2021/08/30 13:21:11.470384 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [PING]
[1] 2021/08/30 13:21:11.470390 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - ->> [PONG]
[1] 2021/08/30 13:21:11.470554 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [SUB _INBOX.hjR8szP0693GIzb0NyDOfK.*  1]
[1] 2021/08/30 13:21:11.470582 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- [PUB $JS.API.INFO _INBOX.hjR8szP0693GIzb0NyDOfK.C9whJU49 0]
[1] 2021/08/30 13:21:11.470592 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <<- MSG_PAYLOAD: [""]
[1] 2021/08/30 13:21:11.470771 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - ->> [MSG _INBOX.hjR8szP0693GIzb0NyDOfK.C9whJU49 1 145]
[1] 2021/08/30 13:21:11.470785 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  580]
[1] 2021/08/30 13:21:11.470803 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"xeFl4pV2T0KFl5y0yE2w5j\",\"timestamp\":\"2021-08-30T13:21:11.470721369Z\",\"server\":\"nats.myserver\",\"client\":{\"start\":\"2021-08-30T13:21:11.469881976Z\",\"host\":\"192.168.240.4\",\"id\":10,\"acc\":\"$G\",\"user\":\"backend\",\"name\":\"backend\",\"lang\":\"go\",\"ver\":\"1.11.0\",\"rtt\":434441,\"server\":\"nats.myserver\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"error\\\":{\\\"code\\\":503,\\\"err_code\\\":10039,\\\"description\\\":\\\"JetStream not enabled for account\\\"}}\"}"]
[1] 2021/08/30 13:21:11.474161 [DBG] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - Client connection closed: Client Closed
[1] 2021/08/30 13:21:11.474185 [TRC] 192.168.240.4:54306 - cid:10 - "v1.11.0:go:backend" - <-> [DELSUB 1]

After this, the service periodically attempts to reconnect but keeps getting the JetStream not enabled for account error over and over.

Interestingly, this doesn't happen all the time. I had to restart the whole setup several times to reproduce this (for 80+% of the attempts it was working as expected). Invoking nats-server --signal reload multiple times after startup seems to fix it, but ofc it's not a long-term solution.

Any ideas on whether my config is incorrect, or if there is some race condition when loading the config (maybe some authorization + accounts inconsistency)?

ripienaar commented 3 years ago

You should probably remove the authorization section you're essentially configuring 2 competing auth systems here, and specifically you have the backend user in both this section and the accounts section.

lorenzodonini commented 3 years ago

@ripienaar I thought it would still be possible to combine authorization-like permission maps with Jetstream. Thanks for the clarification, docs weren't clear about that. Error seems to be gone 😃

ripienaar commented 3 years ago

I believe the account level users still accept authorization blocks too

Zeg0 commented 2 years ago

Maybe try the nsc tool to add a new nats account and user and set jetstream (js) permissions? Worked for me at least...

Like this:

nsc add account MYNATSACCOUNT nsc add user MYNATSUSER export NATS_CA=/path/to/selfsigned/cluster/ca.cer nsc edit account --name MYNATSACCOUNT --js-mem-storeage -1 --js-disk-storage -1 --js-streams -1 --js-consumer -1 nsc push -a MYNATSACCOUNT -u nats://natsclusterurl.com

and finally use the credentials file output in [/unixuserhome/ or root] /.nkeys/creds/... to connect with nats-cluster

nats ..... --creds /path/to/user.creds --tlsca=/path/to/selfsigned/cluster/ca.cer

lionello commented 1 year ago

Seems you'll have to add --js-mem-storage -1 and/or --js-disk-storage -1 when you generate the account JWT.