search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

# can't get jetstream response from leafnode #2161

Closed wchajl closed 3 years ago

wchajl commented 3 years ago

Defect

Make sure that these boxes are checked before submitting your issue -- thank you!

Versions of nats-server and affected client libraries used:

nats-server v2.2.2 natscli development

OS/Container environment:

coud-server : centos-7 leafnode : ubuntu20.1

Steps or code to reproduce the issue:

leafnodes { remotes :[ { url:"nats-leaf://e1:e1@10.110.105.12:7422", } ] }


+ the cloud-server configuration

accounts { CLOUD : { jetstream : enable users : [ {user : cloud,password:cloud} ] } Edge_1 : { users : [ {user:ce1,password:ce1} ] } }

leafnodes { listen : "0.0.0.0:7422" authorization { users =[ {user:c1,password:c1,account:CLOUD} {user:e1,password:e1,account:Edge_1} ] } }

cluster { name : JSC
listen : 0.0.0.0:6222
routes : [ nats-routes://localhost:6222 ] }


1.  creating stream from cloud using account Edge-1
2.  creating consumer from cloud using account Edge-1
3.  query info about stream from cloud using account Edge-1

#### Expected result:
1.  creating stream from cloud using account Edge-1
success and  reply with stream info
2. creating consumer from cloud using account Edge-1
 success  and  reply with consumer info
3.  query info about stream from cloud using account Edge-1
success and get the stream list 

#### Actual result:
1.  creating stream from cloud using account Edge-1
+ the cloud-server

[wangch@host5 test]$ nats -s nats://ce1:ce1@localhost:4222 str add test3 ? Subjects to consume test3 ? Storage backend file ? Retention Policy Work Queue ? Discard Policy New ? Stream Messages Limit -1 ? Message size limit -1 ? Maximum message age limit -1 ? Maximum individual message size -1 ? Duplicate tracking time window 2m ? Replicas 1 nats: error: could not create Stream: JetStream not enabled for account

+ the leafnode

houston@wangchenghao010:~$ [884] 2021/04/26 14:23:41.661636 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LS+ _INBOX.qeQvzjpEQ3LSVxQtXYSkgR.] [884] 2021/04/26 14:23:41.661753 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LMSG $JS.API.STREAM.CREATE.test3 _INBOX.qeQvzjpEQ3LSVxQtXYSkgR.B0qeb3WL 221 ] [884] 2021/04/26 14:23:41.661864 [TRC] 10.110.105.12:7422 - lid:4 - <<- MSG_PAYLOAD: ["{\"name\":\"test3\",\"subjects\":[\"test3\"],\"retention\":\" workqueue\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msg_size\":-1,\"storage\":\"file\",\"discard\":\"new\",\"num_r eplicas\":1,\"duplicate_window\":120000000000}"] [884] 2021/04/26 14:23:41.662665 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LS- _INBOX.qeQvzjpEQ3LSVxQtXYSkgR.] [884] 2021/04/26 14:23:41.662691 [TRC] 10.110.105.12:7422 - lid:4 - <-> [DELSUB _INBOX.qeQvzjpEQ3LSVxQtXYSkgR.*] [884] 2021/04/26 14:23:41.663007 [TRC] 10.110.105.12:7422 - lid:4 - ->> [LS+ test3] [884] 2021/04/26 14:23:41.663206 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1123] [884] 2021/04/26 14:23:41.663296 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"pQEUok29VzIh0gvXE msRAy\",\"timestamp\":\"2021-04-26T06:23:41.66313Z\",\"server\":\"edge1\",\"client\":{\"start\":\"2021-04-26T05:56:34.9492001Z\",\"host\":\"10.110.1 05.12\",\"id\":4,\"acc\":\"$G\",\"rtt\":1446200,\"server\":\"cloud1\"},\"subject\":\"$JS.API.STREAM.CREATE.test3\",\"request\":\"{\\"name\\":\\"t est3\\",\\"subjects\\":[\\"test3\\"],\\"retention\\":\\"workqueue\\",\\"max_consumers\\":-1,\\"max_msgs\\":-1,\\"max_bytes\\":-1,\\" max_age\\":0,\\"max_msg_size\\":-1,\\"storage\\":\\"file\\",\\"discard\\":\\"new\\",\\"num_replicas\\":1,\\"duplicate_window\\":12000 0000000}\",\"response\":\"{\\"type\\":\\"io.nats.jetstream.api.v1.stream_create_response\\",\\"config\\":{\\"name\\":\\"test3\\",\\"subje cts\\":[\\"test3\\"],\\"retention\\":\\"workqueue\\",\\"max_consumers\\":-1,\\"max_msgs\\":-1,\\"max_bytes\\":-1,\\"discard\\":\\"ne w\\",\\"max_age\\":0,\\"max_msg_size\\":-1,\\"storage\\":\\"file\\",\\"num_replicas\\":1,\\"duplicate_window\\":120000000000},\\"creat ed\\":\\"2021-04-26T06:23:41.6622358Z\\",\\"state\\":{\\"messages\\":0,\\"bytes\\":0,\\"first_seq\\":0,\\"first_ts\\":\\"0001-01-01T00 :00:00Z\\",\\"last_seq\\":0,\\"last_ts\\":\\"0001-01-01T00:00:00Z\\",\\"consumer_count\\":0}}\"}"]

+ the leafnode

houston@wangchenghao010:~$ nats str info ? Select a Stream test3 Information for Stream test3 created 2021-04-26T14:23:41+08:00

Configuration:

         Subjects: test3
 Acknowledgements: true
        Retention: File - WorkQueue
         Replicas: 1
   Discard Policy: New
 Duplicate Window: 2m0s
 Maximum Messages: unlimited
    Maximum Bytes: unlimited
      Maximum Age: 0.00s

Maximum Message Size: unlimited Maximum Consumers: unlimited

State:

         Messages: 0
            Bytes: 0 B
         FirstSeq: 0
          LastSeq: 0
 Active Consumers: 0

2.  creating consumer from cloud using account Edge-1
+ the cloud server

[wangch@host5 test]$ nats -s nats://ce1:ce1@localhost:4222 con add test3 test3 ? Delivery target (empty for Pull Consumers) ? Start policy (all, new, last, 1h, msg sequence) all ? Replay policy instant ? Filter Stream by subject (blank for all) ? Maximum Allowed Deliveries -1 ? Maximum Acknowledgements Pending 1 nats: error: could not select Stream: JetStream not enabled for account

+ the leafnode

[884] 2021/04/26 14:25:23.564675 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LS+ _INBOX.2OhM9sXBId9lp3GQu3f5C0.] [884] 2021/04/26 14:25:23.564757 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LMSG $JS.API.CONSUMER.INFO.test3.test3 _INBOX.2OhM9sXBId9lp3GQu3f5C0.nyYIdl EB 0] [884] 2021/04/26 14:25:23.564779 [TRC] 10.110.105.12:7422 - lid:4 - <<- MSG_PAYLOAD: [""] [884] 2021/04/26 14:25:23.565285 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LMSG $JS.API.STREAM.INFO.test3 _INBOX.2OhM9sXBId9lp3GQu3f5C0.6xAIBqbM 0] [884] 2021/04/26 14:25:23.565341 [TRC] 10.110.105.12:7422 - lid:4 - <<- MSG_PAYLOAD: [""] [884] 2021/04/26 14:25:23.565534 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LMSG $JS.API.STREAM.NAMES _INBOX.2OhM9sXBId9lp3GQu3f5C0.1xAbxs2E 12] [884] 2021/04/26 14:25:23.565551 [TRC] 10.110.105.12:7422 - lid:4 - <<- MSG_PAYLOAD: ["{\"offset\":0}"] [884] 2021/04/26 14:25:23.565708 [TRC] 10.110.105.12:7422 - lid:4 - ->> [LMSG _INBOX.2OhM9sXBId9lp3GQu3f5C0.nyYIdlEB 114] [884] 2021/04/26 14:25:23.565791 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 449] [884] 2021/04/26 14:25:23.565826 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"pQEUok29VzIh0gvXE msRNG\",\"timestamp\":\"2021-04-26T06:25:23.5655063Z\",\"server\":\"edge1\",\"client\":{\"start\":\"2021-04-26T05:56:34.9492001Z\",\"host\":\"10.110 .105.12\",\"id\":4,\"acc\":\"$G\",\"rtt\":1446200,\"server\":\"cloud1\"},\"subject\":\"$JS.API.CONSUMER.INFO.test3.test3\",\"response\":\"{\\"type\ \":\\"io.nats.jetstream.api.v1.consumer_info_response\\",\\"error\\":{\\"code\\":404,\\"description\\":\\"consumer not found\\"}}\"}"] [884] 2021/04/26 14:25:23.565858 [TRC] 10.110.105.12:7422 - lid:4 - ->> [LMSG _INBOX.2OhM9sXBId9lp3GQu3f5C0.6xAIBqbM 474] [884] 2021/04/26 14:25:23.565905 [TRC] 10.110.105.12:7422 - lid:4 - ->> [LMSG _INBOX.2OhM9sXBId9lp3GQu3f5C0.1xAbxs2E 126] [884] 2021/04/26 14:25:23.565950 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 481] [884] 2021/04/26 14:25:23.565975 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"pQEUok29VzIh0gvXE msRVS\",\"timestamp\":\"2021-04-26T06:25:23.5656598Z\",\"server\":\"edge1\",\"client\":{\"start\":\"2021-04-26T05:56:34.9492001Z\",\"host\":\"10.110 .105.12\",\"id\":4,\"acc\":\"$G\",\"rtt\":1446200,\"server\":\"cloud1\"},\"subject\":\"$JS.API.STREAM.NAMES\",\"request\":\"{\\"offset\\":0}\",\"r esponse\":\"{\\"type\\":\\"io.nats.jetstream.api.v1.stream_names_response\\",\\"total\\":3,\\"offset\\":0,\\"limit\\":1024,\\"streams\\" :[\\"test\\",\\"test1\\",\\"test3\\"]}\"}"] [884] 2021/04/26 14:25:23.565989 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 853] [884] 2021/04/26 14:25:23.566043 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"pQEUok29VzIh0gvXE msRRM\",\"timestamp\":\"2021-04-26T06:25:23.5656556Z\",\"server\":\"edge1\",\"client\":{\"start\":\"2021-04-26T05:56:34.9492001Z\",\"host\":\"10.110 .105.12\",\"id\":4,\"acc\":\"$G\",\"rtt\":1446200,\"server\":\"cloud1\"},\"subject\":\"$JS.API.STREAM.INFO.test3\",\"response\":\"{\\"type\\":\\" io.nats.jetstream.api.v1.stream_info_response\\",\\"config\\":{\\"name\\":\\"test3\\",\\"subjects\\":[\\"test3\\"],\\"retention\\":\\" workqueue\\",\\"max_consumers\\":-1,\\"max_msgs\\":-1,\\"max_bytes\\":-1,\\"discard\\":\\"new\\",\\"max_age\\":0,\\"max_msg_size\\":- 1,\\"storage\\":\\"file\\",\\"num_replicas\\":1,\\"duplicate_window\\":120000000000},\\"created\\":\\"2021-04-26T06:23:41.6622358Z\\",\ \"state\\":{\\"messages\\":0,\\"bytes\\":0,\\"first_seq\\":0,\\"first_ts\\":\\"0001-01-01T00:00:00Z\\",\\"last_seq\\":0,\\"last_ts\\" :\\"0001-01-01T00:00:00Z\\",\\"consumer_count\\":0}}\"}"] [884] 2021/04/26 14:25:23.566448 [TRC] 10.110.105.12:7422 - lid:4 - <<- [LS- _INBOX.2OhM9sXBId9lp3GQu3f5C0.] [884] 2021/04/26 14:25:23.566497 [TRC] 10.110.105.12:7422 - lid:4 - <-> [DELSUB _INBOX.2OhM9sXBId9lp3GQu3f5C0.*]

+ the leafnode

houston@wangchenghao010:~$ nats con info ? Select a Stream test3 nats: error: could not select Consumer: no Consumers are defined for Stream test3


3.  query info about stream from cloud using account Edge-1
+ the cloud

[wangch@host5 test]$ nats -s nats://ce1:ce1@localhost:4222 str info nats: error: could not pick a Stream to operate on: JetStream not enabled for account

ripienaar commented 3 years ago

Think @matthiashanel was in this code recently and probably know the answer

matthiashanel commented 3 years ago

@wchajl after updating natscli, do you still see the issue? I am having problems reproducing it. (see screenshot)

I also noticed the following: leaf nodes bridge security domains. In your case the the leaf node is running jetstream AND the only account bridged is the global account $G. (That's the account used when there are no accounts listed/no operator is defined) Then the account $G is connected to the account Edge_1. All the jetstream stuff happens in the leaf node. I would avoid connecting different sets of leaf nodes to the same account, as this can result in clashes. (the account Edge_1 essentially seeing multiple jetstreams) Do you want to enable jetstream on your cloud server as well? Also, in your setup, jetstream : enable for the account CLOUD has no effect.

This screen shot shows the server (top) commands (left) and config used (right) Screen Shot 2021-04-26 at 6 01 26 PM

wchajl commented 3 years ago

@matthiashanel sorry, I missed the jetstream :enable server_name : cloud .. fileds in the cloud-server's config . after adding these ,you should reproduce the error as described .

server_name : cloud
port : 4222
jetstream:{
    store_dir : cloud1
}
log_file : cloud

accounts {
    CLOUD : {
        jetstream : enable
        users : [
            {user : cloud,password:cloud}
        ]

    }
    Edge_1 : {
        users : [
            {user:ce1,password:ce1}

        ]

    }

}

leafnodes {
    listen : "0.0.0.0:7422"
    authorization {
        users =[
            {user:c1,password:c1,account:CLOUD}
            {user:e1,password:e1,account:Edge_1}

        ]

    }

}
cluster {
    name : JSC
    listen : 0.0.0.0:6222
    routes : [
        nats-routes://localhost:6222

    ]
}
[26778] 2021/04/27 09:13:45.516820 [INF] Starting nats-server
[26778] 2021/04/27 09:13:45.516866 [INF]   Version:  2.2.2
[26778] 2021/04/27 09:13:45.516869 [INF]   Git:      [not set]
[26778] 2021/04/27 09:13:45.516871 [INF]   Name:     cloud
[26778] 2021/04/27 09:13:45.516877 [INF]   Node:     OgGGxLdl
[26778] 2021/04/27 09:13:45.516880 [INF]   ID:       NBRVHYJBYGMQ4BHTP6ZZF7QVXHLX6NV2MI3XL6AOAVZQJBYIWLSZBDEW
[26778] 2021/04/27 09:13:45.516886 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[26778] 2021/04/27 09:13:45.516889 [INF] Using configuration file: cloud.json
[26778] 2021/04/27 09:13:45.517442 [INF] Starting JetStream
[26778] 2021/04/27 09:13:45.517737 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[26778] 2021/04/27 09:13:45.517744 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[26778] 2021/04/27 09:13:45.517749 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[26778] 2021/04/27 09:13:45.517751 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[26778] 2021/04/27 09:13:45.517753 [INF] 
[26778] 2021/04/27 09:13:45.517756 [INF]          https://docs.nats.io/jetstream
[26778] 2021/04/27 09:13:45.517758 [INF] 
[26778] 2021/04/27 09:13:45.517761 [INF] ---------------- JETSTREAM ----------------
[26778] 2021/04/27 09:13:45.517767 [INF]   Max Memory:      94.23 GB
[26778] 2021/04/27 09:13:45.517772 [INF]   Max Storage:     1.47 TB
[26778] 2021/04/27 09:13:45.517775 [INF]   Store Directory: "cloud1/jetstream"
[26778] 2021/04/27 09:13:45.517777 [INF] -------------------------------------------
[26778] 2021/04/27 09:13:45.518046 [INF] Starting JetStream cluster
[26778] 2021/04/27 09:13:45.518051 [INF] Creating JetStream metadata controller
[26778] 2021/04/27 09:13:45.519957 [INF] JetStream cluster bootstrapping
[26778] 2021/04/27 09:13:45.520512 [INF] Listening for leafnode connections on 0.0.0.0:7422
[26778] 2021/04/27 09:13:45.521766 [INF] Listening for client connections on 0.0.0.0:4222
[26778] 2021/04/27 09:13:45.522396 [INF] Server is ready
[26778] 2021/04/27 09:13:45.522425 [INF] Cluster name is JSC
[26778] 2021/04/27 09:13:45.522460 [INF] Listening for route connections on 0.0.0.0:6222
[26778] 2021/04/27 09:13:45.523246 [INF] [::1]:54367 - rid:8 - Route connection created
[26778] 2021/04/27 09:13:46.521818 [INF] [::1]:54370 - rid:9 - Route connection created
[26778] 2021/04/27 09:14:03.205840 [INF] 10.201.30.78:51480 - lid:10 - Leafnode connection created
[wangch@host5 leaf]$ nats -s nats://ce1:ce1@localhost:4222 str info
nats: error: could not pick a Stream to operate on: JetStream not enabled for account
wchajl commented 3 years ago

@matthiashanel I guess maybe the reason is that when both cloud & leafnode have jetstream enabled ,the client's INBOX would accept the cloud-server's reply first ,and then INBOX subscription shall be closed ? so is it not recommended to have leafnode with jetstream enabled connect to the server with jetstream enabled as well ?

matthiashanel commented 3 years ago

@wchajl, your description matches what's going on. Essentially your account has two jetstreams, which causes issues.

However, it is possible to do that but comes with caveats. If you add a system account to both server and connect them via a leaf node remote entry AND you suppress certain subjects along the leaf node connection, you can create a single jetstream spanning cloud and edge. The downside of this approach is that while a leaf node connection is down, you won't be able to create streams from within edge. Another downside is that your system account would reach all the way to the edbge, so you may have security concerns regarding the ability of edge to retrieve all system account messages (including from cloud). This works today - with some caveats:

  1. Your cloud/edge server need to become a cluster. (There is an issue with cluster of 1. This is very inconvenient and I hope we resolve this quickly)
  2. Account names on either end of the leaf node connection need to be the same. (This is another issue I hope we resolve soon)

This is a set of config files where all of the above is implemented. (start all server-.conf for the cloud and all leaf-.conf for the leaf) JS_over_Leaf.zip

I posted the necessary steps on slack as well.

We are aware for the need of independently operating jetstreams and are actively debating ways to support that.

derekcollison commented 3 years ago

We had basic support for leafnodes but will improve the interop for the next release in a big way. We can tell folks are drawn to JS + leafnode extended clusters.

wchajl commented 3 years ago

@matthiashanel thanks , I've tested your example above .but since we need to create stream in leafnodes from the cloud ,it doesn't satisfied our need now . looking forward to the next release.

derekcollison commented 3 years ago

When having JetStream enabled in only one, that should be well supported. Having it enabled in both is where we will make the improvements.

wchajl commented 3 years ago

@derekcollison thanks, it work.