Messages are not propagated to leaf stream via cross account mirroring

Observed behavior

Given a main and a leaf cluster and a stream on the main cluster. When I create a mirror of that stream and import and export the necessary services with nsc I expect that messages sent and saved in the stream of main cluster are replicated to the stream of the leaf cluster.

Expected behavior

Messages should arrive to the stream on leaf cluster and no errors logged in the server log.

Server and client version

nats-server: v2.10.11 nats: 0.1.1 nsc: 2.8.5

Host environment

$ uname -a
Darwin MacBook-Pro-2.local 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64

vanilla nats-server started as a local executable on localhost

Steps to reproduce

cluster-nsc.tar.gz

Setup nsc environment and users

I used separated nsc environment by setting these up.

export NKEYS_PATH="`pwd`/keys"
nsc env --store "`pwd`/store"

# operator, accounts and users are already created in the tar file
# nsc add operator --generate-signing-key --sys --name main_operator
# nsc edit operator --require-signing-keys --account-jwt-server-url "nats://localhost:4222"

nsc env --operator main_operator

Then I created the users - which you don't need since they are already in the local store - with this:

nsc add account PUBLISHER
nsc edit account PUBLISHER --sk generate
nsc edit account PUBLISHER --js-disk-storage -1 --js-streams -1 --js-consumer -1

nsc add user --account PUBLISHER publisher

nsc add account CONSUMER
nsc edit account CONSUMER --sk generate
nsc edit account CONSUMER --js-disk-storage -1 --js-streams -1 --js-consumer -1

nsc add user --account CONSUMER consumer

The credentials are already generated but you need to create the contexts.

nats context save main_sys --nsc "nsc://main_operator/SYS/sys"
nats context save main_publisher --nsc "nsc://main_operator/PUBLISHER/publisher"
nats context save main_consumer --nsc "nsc://main_operator/CONSUMER/consumer" --server=nats://localhost:4226

Startup

Start the four nats servers in different shells.

./nats-server --config main1.conf --debug
./nats-server --config main2.conf --debug
./nats-server --config leaf1.conf --debug
./nats-server --config leaf21.conf --debug

They create directories in the ./data and *-jwt as full and cache resolvers.

Create and export/import streams

The idea is to create a temperatures stream in the main cluster from temperature.> subject. On the leaf I want to mirror this as leaf-temperatures stream from main.publisher.temperature.> subject.

The two streams are already created as json configs, I put them here.

The source stream on main

{
  "name": "temperatures",
  "subjects": [
    "temperature.\u003e"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": true,
  "mirror_direct": false
}

and the mirrored one.

{
  "name": "leaf-temperatures",
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": 0,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 1,
  "mirror": {
    "name": "temperatures",
    "external": {
      "api": "JS.publisher@main.API",
      "deliver": "main.publisher"
    }
  },
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": true,
  "mirror_direct": false
}

So I create the stream and export the $JS.main API, the flow control and the subject. The I map the subject and the jetstream API and create the leaf stream.

nats --context=main_publisher stream add --config main-stream.json --js-domain main

nsc add export --account PUBLISHER --name API-PUBLISHER --service --response-type Stream --subject '$JS.main.API.CONSUMER.>'
nsc add export --account PUBLISHER --name FC-PUBLISHER --service --response-type Stream --subject '$JS.FC.>'
nsc add export --account PUBLISHER --name Data-PUBLISHER --service --response-type Stream --subject 'temperature.>'

nsc add import --account CONSUMER --src-account PUBLISHER --name Remote-API-PUBLISHER --service --remote-subject '$JS.main.API.CONSUMER.>' --local-subject 'JS.publisher@main.API.CONSUMER.>'
nsc add import --account CONSUMER --src-account PUBLISHER --name Remote-FC-PUBLISHER --service --remote-subject '$JS.FC.>'
nsc add import --account CONSUMER --src-account PUBLISHER --name Remote-Data-PUBLISHER --remote-subject 'temperature.>' --local-subject 'main.publisher.temperature.>'

nsc push -A

nats --context=main_consumer stream add --config leaf-stream.json --js-domain leaf

nats --context=main_publisher publish temperature.UK 'Cold enough'

After publishing into temperatures I cannot see the message in the leaf stream.

 nats --context=main_publisher stream report
Obtaining Stream stats

╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│                                         Stream Report                                         │
├──────────────┬─────────┬───────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream       │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├──────────────┼─────────┼───────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ temperatures │ File    │           │         0 │ 1        │ 55 B  │ 0    │       0 │ main2*   │
╰──────────────┴─────────┴───────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

On the consumer side, leaf side there is no replicate since there are import errors in the logs.

nats --context=main_consumer stream report
Obtaining Stream stats

╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                            Stream Report                                           │
├───────────────────┬─────────┬───────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream            │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├───────────────────┼─────────┼───────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ leaf-temperatures │ File    │           │         0 │ 0        │ 0 B   │ 0    │       0 │ leaf1*   │
╰───────────────────┴─────────┴───────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

╭──────────────────────────────────────────────────────────────────────────────────────────╮
│                                    Replication Report                                    │
├────────┬──────┬────────────┬───────────────┬────────┬─────────────┬────────┬─────┬───────┤
│ Stream │ Kind │ API Prefix │ Source Stream │ Filter │ Destination │ Active │ Lag │ Error │
├────────┼──────┼────────────┼───────────────┼────────┼─────────────┼────────┼─────┼───────┤
╰────────┴──────┴────────────┴───────────────┴────────┴─────────────┴────────┴─────┴───────╯

Errors in all server logs about that the service import is not possible by the CONSUMER account.

[46577] 2024/02/22 15:53:45.661990 [DBG] Adding service import ADDFOWQB2AVHQ7ZZFVV4AY4AFERIQ35FIGILZ6NAKTAFDASLBMB7ZGHU/PUBLISHER:"$JS.FC.>" for ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER:""
[46577] 2024/02/22 15:53:45.661994 [DBG] Error adding service import to account [ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER]: service import not authorized
[46577] 2024/02/22 15:53:45.661997 [DBG] Adding service import ADDFOWQB2AVHQ7ZZFVV4AY4AFERIQ35FIGILZ6NAKTAFDASLBMB7ZGHU/PUBLISHER:"JS.publisher@main.API.CONSUMER.>" for ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER:"$JS.main.API.CONSUMER.>"
[46577] 2024/02/22 15:53:45.662000 [DBG] Error adding service import to account [ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER]: service import not authorized
[46577] 2024/02/22 15:53:45.662003 [DBG] Adding stream import ADDFOWQB2AVHQ7ZZFVV4AY4AFERIQ35FIGILZ6NAKTAFDASLBMB7ZGHU/PUBLISHER:"temperature.>" for ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER:"main.publisher.temperature.>"
[46577] 2024/02/22 15:53:45.662006 [DBG] Error adding stream import to account [ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX/CONSUMER]: stream import not authorized

Not sure what is missing on CONSUMER account or in server config in order that CONSUMER can import those services and streams.

Not sure if there's a mismatch between your copied output when you created the services and when you imported them.

The error from the server seems to indicate that the service exports have a token required. - but when you created them earlier you didn't provide that information - is it possible that tokens are required on the export, but you didn't provide them on the imports - can you nsc describe the accounts

Both configurations should appear on the servers - for starters I would make sure that you are using a mem-resolver that bakes the operator and accounts. I would also reduce the complexity - start with one server (a hub) and add a leaf node to it.

Well we need the nats-resolver since we want to handle nkeys and tokens with nsc. I tried by generating a mem-resolver.conf and use that, but then I failed to push exports and imports from nsc. Also I didn't find anything how can I have an accounts section in case of memory resolver where I can list my exports and import.

What worked

I set up leaf cross account stream mirroring with vanilla nats users and exports and imports in config, like this.

accounts: {
    PUBLISHER: {
        jetstream: enabled
        users: [ { user: publisher, password: password } ]
        exports: [
            { service: "$JS.main.API.CONSUMER.CREATE.*", response_type: "stream" }
            { stream: "temperature.>" }
            { service "$JS.FC.>" }
        ]
    }
    CONSUMER: {
        jetstream: enabled
        users: [ { user: consumer, password: password } ]
        imports: [
            { service: { account: PUBLISHER, subject: "$JS.main.API.CONSUMER.CREATE.*" }, to: "$JS.publisher@main.API.CONSUMER.CREATE.*" }
            { stream: { account: PUBLISHER, subject: "temperature.>" }, to: "main.publisher.temperature.>" }
            { service: { account: PUBLISHER, subject: "$JS.FC.>" } }
        ]
    }
}

And then stream mirroring worked. That is okay, but I still look for the way with pull this through with nsc.

No cluster now

So I switched to single node nats and leaf with the same set of keys. I started main node, did nsc push -A, started leaf which connects back with SYS and CONSUMER to main. And did the same exports and imports and stream creations. The streams have been created although in the DEBUG logs I can see this.

main:

[71131] 2024/02/23 13:06:41.712138 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[71131] 2024/02/23 13:06:41.712167 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[71131] 2024/02/23 13:06:41.712190 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[71131] 2024/02/23 13:06:41.712207 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[71131] 2024/02/23 13:06:41.712211 [DBG] JETSTREAM - JetStream connection closed: Client Closed

leaf:

[71171] 2024/02/23 13:06:23.637388 [DBG] Retrying mirror consumer for 'ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX > leaf-temperatures'
[71171] 2024/02/23 13:06:26.641949 [DBG] Retrying mirror consumer for 'ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX > leaf-temperatures'
[71171] 2024/02/23 13:06:29.646319 [DBG] Retrying mirror consumer for 'ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX > leaf-temperatures'
[71171] 2024/02/23 13:06:32.650100 [DBG] Retrying mirror consumer for 'ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX > leaf-temperatures'

I looked in the traces but I don't know NATS protocol. For me it seems that the internal consumer created by CONSUMER on leaf cannot communicate with the PUBLISHER stream on main.

The describes as you asked.

nsc describe account --name PUBLISHER
+--------------------------------------------------------------------------------------+
|                                   Account Details                                    |
+---------------------------+----------------------------------------------------------+
| Name                      | PUBLISHER                                                |
| Account ID                | ADDFOWQB2AVHQ7ZZFVV4AY4AFERIQ35FIGILZ6NAKTAFDASLBMB7ZGHU |
| Issuer ID                 | OCOWOM3BYQZ4D6Z3RSLS5KPJ5WKMH4J3ADWC7NSF3FJ4IPNKPW5HNZBM |
| Issued                    | 2024-02-22 14:10:52 UTC                                  |
| Expires                   |                                                          |
+---------------------------+----------------------------------------------------------+
| Signing Keys              | AALTKXZQDTEGB43GNHMVSMFYHBUEZGTQ6P2XLW7GQTHNKCNYCCDBSQWR |
+---------------------------+----------------------------------------------------------+
| Max Connections           | Unlimited                                                |
| Max Leaf Node Connections | Unlimited                                                |
| Max Data                  | Unlimited                                                |
| Max Exports               | Unlimited                                                |
| Max Imports               | Unlimited                                                |
| Max Msg Payload           | Unlimited                                                |
| Max Subscriptions         | Unlimited                                                |
| Exports Allows Wildcards  | True                                                     |
| Disallow Bearer Token     | False                                                    |
| Response Permissions      | Not Set                                                  |
+---------------------------+----------------------------------------------------------+
| Jetstream                 | Enabled                                                  |
| Max Disk Storage          | Unlimited                                                |
| Max Mem Storage           | Disabled                                                 |
| Max Streams               | Unlimited                                                |
| Max Consumer              | Unlimited                                                |
| Max Ack Pending           | Consumer Setting                                         |
| Max Ack Pending           | Unlimited                                                |
| Max Bytes                 | optional (Stream setting)                                |
| Max Memory Stream         | Unlimited                                                |
| Max Disk Stream           | Unlimited                                                |
+---------------------------+----------------------------------------------------------+
| Imports                   | None                                                     |
+---------------------------+----------------------------------------------------------+

+------------------------------------------------------------------------------------------------------------------------+
|                                                        Exports                                                         |
+----------------+------------------+-------------------------+------------------------+--------+-------------+----------+
| Name           | Type             | Subject                 | Account Token Position | Public | Revocations | Tracking |
+----------------+------------------+-------------------------+------------------------+--------+-------------+----------+
| FC-PUBLISHER   | Service [Stream] | $JS.FC.>                | -                      | Yes    | 0           | -        |
| API-PUBLISHER  | Service [Stream] | $JS.main.API.CONSUMER.> | -                      | Yes    | 0           | -        |
| Data-PUBLISHER | Service [Stream] | temperature.>           | -                      | Yes    | 0           | -        |
+----------------+------------------+-------------------------+------------------------+--------+-------------+----------+

nsc describe account --name CONSUMER
+--------------------------------------------------------------------------------------+
|                                   Account Details                                    |
+---------------------------+----------------------------------------------------------+
| Name                      | CONSUMER                                                 |
| Account ID                | ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX |
| Issuer ID                 | OCOWOM3BYQZ4D6Z3RSLS5KPJ5WKMH4J3ADWC7NSF3FJ4IPNKPW5HNZBM |
| Issued                    | 2024-02-23 12:06:20 UTC                                  |
| Expires                   |                                                          |
+---------------------------+----------------------------------------------------------+
| Signing Keys              | ABRPF6LM3WDSDDSSRWFDTQ2VPICBABRRVJGCJHW3USCAUL4XSYHQR532 |
+---------------------------+----------------------------------------------------------+
| Max Connections           | Unlimited                                                |
| Max Leaf Node Connections | Unlimited                                                |
| Max Data                  | Unlimited                                                |
| Max Exports               | Unlimited                                                |
| Max Imports               | Unlimited                                                |
| Max Msg Payload           | Unlimited                                                |
| Max Subscriptions         | Unlimited                                                |
| Exports Allows Wildcards  | True                                                     |
| Disallow Bearer Token     | False                                                    |
| Response Permissions      | Not Set                                                  |
+---------------------------+----------------------------------------------------------+
| Jetstream                 | Enabled                                                  |
| Max Disk Storage          | Unlimited                                                |
| Max Mem Storage           | Disabled                                                 |
| Max Streams               | Unlimited                                                |
| Max Consumer              | Unlimited                                                |
| Max Ack Pending           | Consumer Setting                                         |
| Max Ack Pending           | Unlimited                                                |
| Max Bytes                 | optional (Stream setting)                                |
| Max Memory Stream         | Unlimited                                                |
| Max Disk Stream           | Unlimited                                                |
+---------------------------+----------------------------------------------------------+
| Exports                   | None                                                     |
+---------------------------+----------------------------------------------------------+

+--------------------------------------------------------------------------------------------------------------------------------+
|                                                            Imports                                                             |
+-----------------------+---------+-------------------------+----------------------------------+---------+--------------+--------+
| Name                  | Type    | Remote                  | Local                            | Expires | From Account | Public |
+-----------------------+---------+-------------------------+----------------------------------+---------+--------------+--------+
| Remote-FC-PUBLISHER   | Service | $JS.FC.>                |                                  |         | PUBLISHER    | Yes    |
| Remote-API-PUBLISHER  | Service | $JS.main.API.CONSUMER.> | JS.publisher@main.API.CONSUMER.> |         | PUBLISHER    | Yes    |
| Remote-Data-PUBLISHER | Stream  | temperature.>           | main.publisher.temperature.>     |         | PUBLISHER    | Yes    |
+-----------------------+---------+-------------------------+----------------------------------+---------+--------------+--------+

Tracked the limitation that is preventing your example from working, and did a write up on it here: https://gist.github.com/aricart/50607a7b5ed0150aaa163f0bfe64f72a

The TDLR; is that you cannot use the same authentication on both the leaf node and the hub server (you had already ran into that as well) - You can use nsc to do the updates, but the operators/accounts/users should be different. If you reuse the operator hierarchy between the two servers the source for the mirror cannot be found. I will discuss with @derekcollison to ensure this is not a bug

The writeup greatly simplifies what the task is - in your example you are creating a fairly complex setup that is possibly closer to reality but when trying to work some of these things we all benefit from simpler setups.

Thanks for the write-up, it is very useful, I didn't check the consumer part since I am using only stream mirroring. The token position part was surprising, later when I looked for that in the docs.nats.io I found only one reference to it in the subject mapping page. But it seems that it is quite important since it authorized the import - that is why I got 'service import not authorized' error.

So then I simplified the example, but still going with nats-resolver on hub and cache nats-resolver on leaf, I managed to mirror the streams.

nsc add export --account PUBLISHER --service --subject '$JS.main.API.CONSUMER.>'
nsc add export --account PUBLISHER --service --subject '$JS.FC.>'
nsc add export --account PUBLISHER --subject 'toaccount.*.>' --account-token-position 2

nsc add import --account CONSUMER --src-account PUBLISHER --service --remote-subject '$JS.main.API.>' --local-subject '$JS.publisher.API.>'
nsc add import --account CONSUMER --src-account PUBLISHER --service --remote-subject '$JS.FC.>'
# this is the key of the CONSUMER account, so the importing account
nsc add import --account CONSUMER --src-account PUBLISHER --remote-subject 'toaccount.ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX.>'

nsc push -A

nats --context=main_consumer --js-domain leaf stream add temperatue_on_leaf --mirror temperatures
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name publisher
? Delivery prefix toaccount.ADTB2JW5LLXDSGZY2FNV4IM4LGIUKWGTB4XDDFXBGOJJR3XTKL5KPVAX

So it managed to mirrored the messages from the main/hub stream to the leaf. Though it would be good to see this documented, because it seems that this importing by key and key-match is a quite important restriction in NATS.

I started to track back from where that import-error can come and I found import check in accounts.go https://github.com/nats-io/nats-server/blob/2cbbc070f33492af13c8322c2885849929fea0ed/server/accounts.go#L2515 and I found that this line checks the equality of the keys (account.Name in the source). https://github.com/nats-io/nats-server/blob/2cbbc070f33492af13c8322c2885849929fea0ed/server/accounts.go#L2634

So somehow it would be good to be referred in the documentation - probably in the nsc export/import part.

mirroring. The token position part was surprising, later when I looked for that in the docs.nats.io I found only one reference to it in the subject mapping page. But it seems that it is quite important since it authorized the import - that is why I got 'service import not authorized' error.

Actually no, I think you provided the --private flag, in that case you'll need to provide an activation to the import that authorizes the account to use your export.

The account_token_position allows a public service to identify the source since that token in the subject can only be set to the account of the client accessing it - thus you know they originate in the account.

I know there was a writeup on the account token position because I wrote it. Not sure what happened.

Regardless - if you didn't specify that option on the service or the stream, the service/stream would be possible. I do believe that the permission error was because at some point the export was private.

Well explicitly I didn't export subjects and services as private but maybe I missed something which made the export worked as a private export. I am investigating still, I come back soon with some results.

I think now I get it, let me summarize what I learned.

The most important is the mental model of stream mirroring. During mirroring internally there is a consumer made which forwards the messages from the stream to a subject, this is what deliver prefix means. So we need to export this prefix or some more generic subject path in order that one - publicly - can import that subject. Here we shouldn't make a local subject mapping during import since it messes up the internal consumer delivery subject. My mistake was to believe that that subject export had anything to do with the stream's subject. No.
It was a very good idea of your that we should expose foreign jetstream API no like JS.user@domain.API but $JS.user_domain.API, so as a new made-up domain. Though several video tutorial does the first one, I prefer this later solution.
Flow control import/export is per domain. In the examples we always use hub or no specific domain, so $JS.FC.> are imported as is. Although in your example the exposed made-up domain is A, so we export $JS.A.API.> and $JS.A.FC.>. If I didn't export the domain's flow control I ran in the same problem someone else already reported here, the mirroring had huge lag although it was slowly catching up.
Also FC is underdocumented, I found only one hit in docs.nats.io on that saying pretty less.
During nsc import in the --src-account I need to use the public key of the exporting account although in an nsc command the account name is also fine (I guess it makes a lookup for the public key and replaces it in the imports).

After all of this the export/import as the following.

nsc add export --account PUBLISHER --service --subject '$JS.main.API.CONSUMER.>'
nsc add export --account PUBLISHER --service --subject '$JS.main.FC.>'
# Generic subject export for multiple possible stream mirrors
nsc add export --account PUBLISHER --subject 'to_leaf.>'

nsc add import --account CONSUMER --src-account PUBLISHER --service --remote-subject '$JS.main.API.>' --local-subject '$JS.publisher.API.>'
nsc add import --account CONSUMER --src-account PUBLISHER --service --remote-subject '$JS.main.FC.>' --local-subject '$JS.publisher.FC.>'
# during stream mirroring this is the delivery prefix
nsc add import --account CONSUMER --src-account PUBLISHER --remote-subject 'to_leaf.temperatures.>'

Now we can make a mirror from foreign domain publisher and delivery prefix to_leaf.temperatures.

Since concepts have been clear up, I close this issue.

nats-io / nats-server