search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

Leaf node jetstream not receiving messages after restart #3024

Closed JH7 closed 2 years ago

JH7 commented 2 years ago

Defect

Description

In our production environment, we use two instances of nats servers. One instance is the main server (accessible only from within our Kubernetes cluster) while the other one is a leaf node. The leaf node is accessible from outside of our Kubernetes cluster (via websocket).

We use accounts to isolate and manage messages and topics between the main server and the leaf node. The main server account DMZ exports a stream which the leaf node account CLIENT imports. Note that there is another account, PUBLIC, on both main and leaf server which is used as authorization account (as shown in the configs below). Furthermore, we have a jetstream on this exported stream on the leaf node's CLIENT account. This configuration does work (the jetstream shows messages when they are published on the main server) on the first start of the instances, while it breaks whenever the instances are restarted. Messages published on the main server's DMZ account are not reaching the CLIENT's jetstream on the leaf node anymore. Interestingly, by manually subscribing to the jetstream's topic using nats --server localhost:4111 --user client --password client sub 'orders.>', the messages reach the jetstream again. As soon as the subscription is terminated, the messages are not reaching the jetstream again. Note that this issue occured in our production environment first but I was able to replicate this issue on my local machine.

My guess is that some internal routing tables (in main or leaf server, or both?) are not updated when the leaf node restarts. The log does not show the re-importing of subscriptions after the leaf node has restarted.

I was not able to fix this without removing and re-creating the jetstream.

Versions of nats-server and affected client libraries used:

nats-server: 2.7.4 nast-cli: 0.0.28

OS/Container environment (used in the reproduction):

Windows 10 (WSL2) 4.19.128-microsoft-standard; Ubuntu 20.04.3 LTS Docker Desktop 4.3.2, Engine 20.10.11, Compose 1.29.2

Steps or code to reproduce the issue:

Main server config:

jetstream {
  domain: hub
}

monitor_port: 8222

leafnodes {
  port: 7422
  no_advertise: true
  authorization: {
    user: leaf
    password: leaf
    account: PUBLIC
  }
}

accounts: {
  DMZ: {
    users = [
      { user: service, password: service}
    ]
    exports: [
      {stream: orders.*.stream.>}
    ]
    jetstream: enabled
  }

  PUBLIC: {
    imports: [
      { stream: { account: DMZ, subject: orders.*.stream.>}}
    ]
  }
}

Leaf node config:

jetstream {
  domain: leaf
}

monitor_port: 8222

websocket {
  listen: "0.0.0.0:4112"
  no_tls: true
  compression: false
}

listen: "0.0.0.0:4111"
leafnodes {
  remotes = [
    {
      url: "nats-leaf://leaf:leaf@nats.local:7422"
      account: PUBLIC
    }
  ]
}

accounts: {
  PUBLIC: {
    exports: [
      { stream: orders.*.stream.> }
    ]
  },
  CLIENT: {
    users:[{user: client, password: client}]
    imports: [
      { stream: { account: PUBLIC, subject: orders.client.stream.> }}
    ]
    jetstream: enabled
  },
}

Docker compose file used:

version: '3.5'
services:
  nats-base:
    image: nats
    ports:
      - "4222:4222"
      - "8222:8222"
      - "7422:7422"
    volumes:
      - ./main_test.conf:/etc/nats/nats-server.conf:ro
    command: -c /etc/nats/nats-server.conf -DV
  nats-leaf:
    image: nats
    ports:
      - "4111:4111"
      - "4112:4112"
      - "8223:8222"
    volumes:
      - ./leaf_test.conf:/etc/nats/nats-server.conf:ro
    command: -c /etc/nats/nats-server.conf -DV
    depends_on:
      - nats-base
    links:
      - "nats-base:nats.local"
  1. Spin up two nats-server instances with the configurations above. One should be the "main" server and the other one should be configured as a leaf node.
  2. Create a stream within the CLIENT account on the leaf node: nats --server localhost:4111 --user client --password client stream create orders '--subjects=orders.*.stream.entry' --retention=work --max-consumers=-1 --max-msgs-per-subject=-1 --max-msgs=-1 --max-bytes=-1 --max-age=-1 --max-msg-size=-1 --storage=file --discard=old --replicas=1 --dupe-window="2m0s" --no-allow-rollup --no-deny-delete --no-deny-purge
  3. Publish a message on the main server using nats --user service --password service pub 'orders.client.stream.entry' 'test1'
  4. Verify that the stream contains one message using nats --server localhost:4111 --user client --password client stream info orders
  5. Now, restart the leaf node. I did that via the Docker Desktop GUI.
  6. Try publishing a message on the main server again using nats --user service --password service pub 'orders.client.stream.entry' 'test2'

Expected result:

The stream contains the published message.

Actual result:

The stream does not contain the published message. Verify this by using nats --server localhost:4111 --user client --password client stream info orders again. It does not show two messages.

Log

dev-nats-base-1  | [1] 2022/04/11 14:54:58.544648 [INF] Starting nats-server
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544679 [INF]   Version:  2.7.4
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544681 [INF]   Git:      [a86b84a]
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544683 [DBG]   Go build: go1.17.8
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544684 [INF]   Name:     NC5AELZGBN5SRQJFQLJQVJ4LWOSPWWB5KUHAP36HHYUXBWLDWBLDLB2V
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544686 [INF]   Node:     nU3YjYyc
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544687 [INF]   ID:       NC5AELZGBN5SRQJFQLJQVJ4LWOSPWWB5KUHAP36HHYUXBWLDWBLDLB2V
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544689 [WRN] Plaintext passwords detected, use nkeys or bcrypt
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544695 [INF] Using configuration file: /etc/nats/nats-server.conf
dev-nats-base-1  | [1] 2022/04/11 14:54:58.544705 [DBG] Created system account: "$SYS"
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545042 [INF] Starting http monitor on 0.0.0.0:8222
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545077 [INF] Starting JetStream
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545215 [DBG] JetStream creating dynamic configuration - 9.32 GB memory, 144.53 GB disk
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545297 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545309 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545311 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545312 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545313 [INF] 
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545314 [INF]          https://docs.nats.io/jetstream
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545315 [INF] 
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545316 [INF] ---------------- JETSTREAM ----------------
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545318 [INF]   Max Memory:      9.32 GB
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545319 [INF]   Max Storage:     144.53 GB
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545321 [INF]   Store Directory: "/tmp/nats/jetstream"
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545322 [INF]   Domain:          hub
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545323 [INF] -------------------------------------------
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545354 [DBG]   Exports:
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545365 [DBG]      $JS.API.>
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545450 [DBG] Enabled JetStream for account "DMZ"
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545461 [DBG]   Max Memory:      -1 B
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545462 [DBG]   Max Storage:     -1 B
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545571 [DBG] JetStream state for account "DMZ" recovered
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545640 [INF] Listening for leafnode connections on 0.0.0.0:7422
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545652 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545751 [DBG]   ip=172.19.0.2
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545868 [INF] Listening for client connections on 0.0.0.0:4222
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545879 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545938 [DBG]   ip=172.19.0.2
dev-nats-base-1  | [1] 2022/04/11 14:54:58.545970 [INF] Server is ready
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446285 [INF] 172.19.0.3:42978 - lid:6 - Leafnode connection created 
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446612 [TRC] 172.19.0.3:42978 - lid:6 - <<- [CONNECT {"user":"leaf","pass":"[REDACTED]","tls_required":false,"server_id":"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS","domain":"leaf","name":"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS","headers":true}]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446688 [INF] 172.19.0.3:42978 - lid:6 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446715 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446727 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.META.>" -> $JS.API.META.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446731 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446741 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.$KV.>" -> $KV.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446745 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446748 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.INFO" -> $JS.API.INFO to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446752 [INF] 172.19.0.3:42978 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446756 [INF] 172.19.0.3:42978 - lid:6 - Adding deny "$JS.hub.API.>" for outgoing messages to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446773 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.$KV.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446775 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.$OBJ.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446777 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.INFO]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446779 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.STREAM.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446780 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.CONSUMER.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446782 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.META.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446784 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446785 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.API.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446787 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $LDS.snwsb54Z6k3qSK5oqg3zU9]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446789 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.446802 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LS+ $JS.hub.API.SERVER.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447145 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.INFO]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447172 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.STREAM.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447176 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.CONSUMER.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447178 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447182 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.SERVER.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447194 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.$KV.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447197 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $LDS.4CoZJI0ZS86jV7ukmz8mla]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447200 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447211 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.META.>]
dev-nats-base-1  | [1] 2022/04/11 14:54:59.447215 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ $JS.leaf.API.$OBJ.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:00.463767 [DBG] 172.19.0.3:42978 - lid:6 - Leafnode Ping Timer
dev-nats-base-1  | [1] 2022/04/11 14:55:00.463799 [TRC] 172.19.0.3:42978 - lid:6 - ->> [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:00.463972 [TRC] 172.19.0.3:42978 - lid:6 - <<- [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:00.524291 [TRC] 172.19.0.3:42978 - lid:6 - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:00.524308 [TRC] 172.19.0.3:42978 - lid:6 - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:03.418074 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS+ orders.*.stream.entry]
dev-nats-base-1  | [1] 2022/04/11 14:55:03.418097 [DBG] 172.19.0.3:42978 - lid:6 - Creating import subscription on "orders.*.stream.entry" from account "DMZ"
dev-nats-base-1  | [1] 2022/04/11 14:55:10.493729 [DBG] 172.19.0.1:59438 - cid:7 - Client connection created
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494176 [TRC] 172.19.0.1:59438 - cid:7 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"service","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494230 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494240 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494585 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB orders.client.stream.entry 5]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494600 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: ["test1"]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494618 [TRC] 172.19.0.3:42978 - lid:6 - ->> [LMSG orders.client.stream.entry 5]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494628 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.494630 [TRC] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:10.495049 [DBG] 172.19.0.1:59438 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104119 [TRC] 172.19.0.3:42978 - lid:6 - <<- [LS- orders.*.stream.entry]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104134 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB orders.*.stream.entry]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104506 [INF] 172.19.0.3:42978 - lid:6 - Leafnode connection closed: Client Closed account: PUBLIC
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104557 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.STREAM.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104575 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.CONSUMER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104578 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104582 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.SERVER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104594 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.$KV.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104613 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104618 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.META.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104622 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.INFO]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104626 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $JS.leaf.API.$OBJ.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:17.104703 [TRC] 172.19.0.3:42978 - lid:6 - <-> [DELSUB $LDS.4CoZJI0ZS86jV7ukmz8mla]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965052 [INF] 172.19.0.3:43004 - lid:8 - Leafnode connection created 
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965348 [TRC] 172.19.0.3:43004 - lid:8 - <<- [CONNECT {"user":"leaf","pass":"[REDACTED]","tls_required":false,"server_id":"NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX","domain":"leaf","name":"NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX","headers":true}]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965392 [INF] 172.19.0.3:43004 - lid:8 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965413 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.META.>" -> $JS.API.META.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965418 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965422 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.$KV.>" -> $KV.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965425 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965427 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.INFO" -> $JS.API.INFO to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965431 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965444 [INF] 172.19.0.3:43004 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965448 [INF] 172.19.0.3:43004 - lid:8 - Adding deny "$JS.hub.API.>" for outgoing messages to account "PUBLIC"
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965462 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.API.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965468 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965470 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.$OBJ.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965472 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.INFO]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965474 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965475 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.SERVER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965477 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.$KV.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965479 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.STREAM.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965480 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.CONSUMER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965482 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $JS.hub.API.META.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965484 [TRC] 172.19.0.3:43004 - lid:8 - ->> [LS+ $LDS.snwsb54Z6k3qSK5oqg3zU9]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965763 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $LDS.8mRek9GPVzK5CbybhoOS4b]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965777 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.$OBJ.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965782 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.STREAM.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965785 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.$KV.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965797 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.INFO]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965811 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.CONSUMER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965814 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.META.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965817 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $JS.leaf.API.SERVER.>]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965820 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:20.965825 [TRC] 172.19.0.3:43004 - lid:8 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-base-1  | [1] 2022/04/11 14:55:22.067457 [DBG] 172.19.0.3:43004 - lid:8 - Leafnode Ping Timer
dev-nats-base-1  | [1] 2022/04/11 14:55:22.067498 [TRC] 172.19.0.3:43004 - lid:8 - ->> [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:22.067684 [TRC] 172.19.0.3:43004 - lid:8 - <<- [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:22.067694 [TRC] 172.19.0.3:43004 - lid:8 - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:22.067696 [TRC] 172.19.0.3:43004 - lid:8 - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.229925 [DBG] 172.19.0.1:59452 - cid:9 - Client connection created
dev-nats-base-1  | [1] 2022/04/11 14:55:25.230832 [TRC] 172.19.0.1:59452 - cid:9 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"service","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.230898 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.230908 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.231289 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB orders.client.stream.entry 5]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.231305 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: ["test2"]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.231311 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.231313 [TRC] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-base-1  | [1] 2022/04/11 14:55:25.231626 [DBG] 172.19.0.1:59452 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed

dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444053 [INF] Starting nats-server
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444135 [INF]   Version:  2.7.4
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444137 [INF]   Git:      [a86b84a]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444139 [DBG]   Go build: go1.17.8
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444140 [INF]   Name:     NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444143 [INF]   Node:     d3Z34jgW
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444144 [INF]   ID:       NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444146 [WRN] Plaintext passwords detected, use nkeys or bcrypt
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444167 [INF] Using configuration file: /etc/nats/nats-server.conf
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444180 [DBG] Created system account: "$SYS"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444610 [INF] Starting http monitor on 0.0.0.0:8222
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444644 [INF] Starting JetStream
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444763 [DBG] JetStream creating dynamic configuration - 9.32 GB memory, 144.53 GB disk
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444851 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444863 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444865 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444866 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444867 [INF] 
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444868 [INF]          https://docs.nats.io/jetstream
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444869 [INF] 
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444870 [INF] ---------------- JETSTREAM ----------------
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444872 [INF]   Max Memory:      9.32 GB
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444873 [INF]   Max Storage:     144.53 GB
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444874 [INF]   Store Directory: "/tmp/nats/jetstream"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444875 [INF]   Domain:          leaf
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444876 [INF] -------------------------------------------
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444905 [DBG]   Exports:
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.444914 [DBG]      $JS.API.>
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445022 [DBG] Enabled JetStream for account "CLIENT"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445037 [DBG]   Max Memory:      -1 B
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445039 [DBG]   Max Storage:     -1 B
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445392 [DBG] JetStream state for account "CLIENT" recovered
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445456 [INF] Listening for websocket clients on ws://0.0.0.0:4112
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445467 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445470 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445575 [DBG]   ip=172.19.0.3
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445617 [INF] Listening for client connections on 0.0.0.0:4111
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445620 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445672 [DBG]   ip=172.19.0.3
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.445694 [INF] Server is ready
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446102 [DBG] Trying to connect as leafnode to remote server on "nats.local:7422" (172.19.0.2:7422)
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446267 [INF] 172.19.0.2:7422 - lid:6 - Leafnode connection created for account: PUBLIC 
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446467 [DBG] 172.19.0.2:7422 - lid:6 - Remote leafnode connect msg sent
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446898 [INF] 172.19.0.2:7422 - lid:6 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446923 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.META.>" -> $JS.API.META.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446927 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446930 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.$KV.>" -> $KV.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446934 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446936 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.INFO" -> $JS.API.INFO to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446943 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446947 [INF] 172.19.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446960 [INF] 172.19.0.2:7422 - lid:6 - Adding deny "$JS.leaf.API.>" for outgoing messages to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446964 [DBG] 172.19.0.2:7422 - lid:6 - Not permitted to import service "$JS.API.>" on behalf of PUBLIC
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446977 [DBG] 172.19.0.2:7422 - lid:6 - Not permitted to subscribe to "$JS.API.>" on behalf of PUBLIC
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446980 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.INFO]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446982 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.STREAM.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446985 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.CONSUMER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446993 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446995 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.SERVER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446997 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.$KV.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.446999 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $LDS.4CoZJI0ZS86jV7ukmz8mla]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447001 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447002 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.META.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447004 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.$OBJ.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447016 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.$KV.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447027 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.$OBJ.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447029 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.INFO]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447043 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.STREAM.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447045 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.CONSUMER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447049 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.META.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447051 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447056 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.API.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447079 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $LDS.snwsb54Z6k3qSK5oqg3zU9]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447090 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:54:59.447096 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.SERVER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:00.463894 [TRC] 172.19.0.2:7422 - lid:6 - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:00.463905 [TRC] 172.19.0.2:7422 - lid:6 - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:00.524147 [DBG] 172.19.0.2:7422 - lid:6 - Leafnode Ping Timer
dev-nats-leaf-1  | [1] 2022/04/11 14:55:00.524188 [TRC] 172.19.0.2:7422 - lid:6 - ->> [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:00.524394 [TRC] 172.19.0.2:7422 - lid:6 - <<- [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.415780 [DBG] 172.19.0.1:42780 - cid:7 - Client connection created
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.416226 [TRC] 172.19.0.1:42780 - cid:7 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.416299 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.416309 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.417332 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [SUB _INBOX.mh3KRbojJSS3qO6AdoM1SX.*  1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.417345 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB $JS.API.STREAM.CREATE.orders _INBOX.mh3KRbojJSS3qO6AdoM1SX.g1oJNq1s 344]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.417357 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: ["{\"name\":\"orders\",\"subjects\":[\"orders.*.stream.entry\"],\"retention\":\"workqueue\",\"max_consumers\":-1,\"max_msgs_per_subject\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msg_size\":-1,\"storage\":\"file\",\"discard\":\"old\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"sealed\":false,\"deny_delete\":false,\"deny_purge\":false,\"allow_rollup_hdrs\":false}"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.417878 [DBG] JETSTREAM - Creating import subscription on "orders.*.stream.entry" from account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.417893 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS+ orders.*.stream.entry]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.418015 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [MSG _INBOX.mh3KRbojJSS3qO6AdoM1SX.g1oJNq1s 1 617]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.418040 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1642]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.418056 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"4CoZJI0ZS86jV7ukmz8myt\",\"timestamp\":\"2022-04-11T14:55:03.4179786Z\",\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"client\":{\"start\":\"2022-04-11T14:55:03.4157407Z\",\"host\":\"172.19.0.1\",\"id\":7,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.28\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":496500,\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.CREATE.orders\",\"request\":\"{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs_per_subject\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"discard\\\":\\\"old\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_create_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-11T14:55:03.4175439Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"did_create\\\":true}\",\"domain\":\"leaf\"}"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.420296 [DBG] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:03.420325 [TRC] 172.19.0.1:42780 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.28" - <-> [DELSUB 1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.158533 [DBG] 172.19.0.1:42784 - cid:12 - Client connection created
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.158965 [TRC] 172.19.0.1:42784 - cid:12 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159021 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159027 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159301 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [SUB _INBOX.b1hl73jDSy7kanBRfqaHJb.*  1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159321 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB $JS.API.STREAM.INFO.orders _INBOX.b1hl73jDSy7kanBRfqaHJb.fsRWd9qA 0]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159325 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: [""]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159458 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [MSG _INBOX.b1hl73jDSy7kanBRfqaHJb.fsRWd9qA 1 613]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159472 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1238]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.159484 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"4CoZJI0ZS86jV7ukmz8n3K\",\"timestamp\":\"2022-04-11T14:55:07.1594322Z\",\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"client\":{\"start\":\"2022-04-11T14:55:07.1584862Z\",\"host\":\"172.19.0.1\",\"id\":12,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.28\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":491200,\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.orders\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-11T14:55:03.4175439Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"domain\\\":\\\"leaf\\\"}\",\"domain\":\"leaf\"}"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.161804 [DBG] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:07.161833 [TRC] 172.19.0.1:42784 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.28" - <-> [DELSUB 1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:10.494708 [TRC] 172.19.0.2:7422 - lid:6 - <<- [LMSG orders.client.stream.entry 5]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:10.494730 [TRC] 172.19.0.2:7422 - lid:6 - <<- MSG_PAYLOAD: ["test1"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916149 [DBG] 172.19.0.1:42792 - cid:13 - Client connection created
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916599 [TRC] 172.19.0.1:42792 - cid:13 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916651 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916661 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916968 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [SUB _INBOX.OgqVU7byJZN0PkGYCckewu.*  1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916989 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB $JS.API.STREAM.INFO.orders _INBOX.OgqVU7byJZN0PkGYCckewu.wgUwht6m 0]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.916994 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: [""]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.917095 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [MSG _INBOX.OgqVU7byJZN0PkGYCckewu.wgUwht6m 1 647]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.917109 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1274]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.917130 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"4CoZJI0ZS86jV7ukmz8n7l\",\"timestamp\":\"2022-04-11T14:55:11.9170667Z\",\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"client\":{\"start\":\"2022-04-11T14:55:11.9161105Z\",\"host\":\"172.19.0.1\",\"id\":13,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.28\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":500100,\"server\":\"NBNVT6GVSBJ464DTGN7YN7OTDDLEHLEMHRZ26D2EFD6UC55AE3HM4TVS\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.orders\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-11T14:55:03.4175439Z\\\",\\\"state\\\":{\\\"messages\\\":1,\\\"bytes\\\":61,\\\"first_seq\\\":1,\\\"first_ts\\\":\\\"2022-04-11T14:55:10.4947619Z\\\",\\\"last_seq\\\":1,\\\"last_ts\\\":\\\"2022-04-11T14:55:10.4947619Z\\\",\\\"num_subjects\\\":1,\\\"consumer_count\\\":0},\\\"domain\\\":\\\"leaf\\\"}\",\"domain\":\"leaf\"}"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.919661 [DBG] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:11.919690 [TRC] 172.19.0.1:42792 - cid:13 - "v1.13.0:go:NATS CLI Version 0.0.28" - <-> [DELSUB 1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103780 [DBG] Trapped "terminated" signal
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103862 [INF] Initiating Shutdown...
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103872 [INF] Initiating JetStream Shutdown...
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103897 [TRC] JETSTREAM - <-> [DELSUB 1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103928 [TRC] 172.19.0.2:7422 - lid:6 - ->> [LS- orders.*.stream.entry]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103970 [DBG] JETSTREAM - JetStream connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.103976 [DBG] JETSTREAM - JetStream connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104042 [DBG] JETSTREAM - JetStream connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104212 [INF] JetStream Shutdown
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104340 [DBG] Client accept loop exiting..
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104341 [INF] 172.19.0.2:7422 - lid:6 - Leafnode connection closed: Server Shutdown account: PUBLIC
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104384 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.$KV.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104389 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.$OBJ.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104392 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.CONSUMER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104395 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.META.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104399 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.API.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104402 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $LDS.snwsb54Z6k3qSK5oqg3zU9]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104405 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104408 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.SERVER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104411 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.INFO]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104414 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.STREAM.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104417 [TRC] 172.19.0.2:7422 - lid:6 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104353 [DBG] SYSTEM - System connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:17.104505 [INF] Server Exiting..
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962932 [INF] Starting nats-server
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962955 [INF]   Version:  2.7.4
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962956 [INF]   Git:      [a86b84a]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962958 [DBG]   Go build: go1.17.8
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962960 [INF]   Name:     NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962969 [INF]   Node:     YI420yud
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962971 [INF]   ID:       NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962973 [WRN] Plaintext passwords detected, use nkeys or bcrypt
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.962979 [INF] Using configuration file: /etc/nats/nats-server.conf
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963005 [DBG] Created system account: "$SYS"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963366 [INF] Starting http monitor on 0.0.0.0:8222
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963397 [INF] Starting JetStream
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963422 [DBG] JetStream creating dynamic configuration - 9.32 GB memory, 144.53 GB disk
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963554 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963567 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963568 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963569 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963570 [INF] 
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963571 [INF]          https://docs.nats.io/jetstream
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963571 [INF] 
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963572 [INF] ---------------- JETSTREAM ----------------
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963575 [INF]   Max Memory:      9.32 GB
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963576 [INF]   Max Storage:     144.53 GB
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963577 [INF]   Store Directory: "/tmp/nats/jetstream"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963578 [INF]   Domain:          leaf
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963579 [INF] -------------------------------------------
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963606 [DBG]   Exports:
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963611 [DBG]      $JS.API.>
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963700 [DBG] Enabled JetStream for account "CLIENT"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963712 [DBG]   Max Memory:      -1 B
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963714 [DBG]   Max Storage:     -1 B
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.963728 [DBG] Recovering JetStream state for account "CLIENT"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964110 [DBG] JETSTREAM - Creating import subscription on "orders.*.stream.entry" from account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964148 [INF]   Restored 1 messages for stream 'CLIENT > orders'
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964184 [DBG] JetStream state for account "CLIENT" recovered
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964257 [INF] Listening for websocket clients on ws://0.0.0.0:4112
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964268 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964270 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964359 [DBG]   ip=172.19.0.3
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964402 [INF] Listening for client connections on 0.0.0.0:4111
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964415 [DBG] Get non local IPs for "0.0.0.0"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964482 [DBG]   ip=172.19.0.3
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964508 [INF] Server is ready
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.964852 [DBG] Trying to connect as leafnode to remote server on "nats.local:7422" (172.19.0.2:7422)
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965029 [INF] 172.19.0.2:7422 - lid:9 - Leafnode connection created for account: PUBLIC 
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965240 [DBG] 172.19.0.2:7422 - lid:9 - Remote leafnode connect msg sent
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965559 [INF] 172.19.0.2:7422 - lid:9 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965582 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.INFO" -> $JS.API.INFO to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965589 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965593 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965624 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.META.>" -> $JS.API.META.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965633 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965646 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.$KV.>" -> $KV.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965650 [INF] 172.19.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965655 [INF] 172.19.0.2:7422 - lid:9 - Adding deny "$JS.leaf.API.>" for outgoing messages to account "PUBLIC"
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965667 [DBG] 172.19.0.2:7422 - lid:9 - Not permitted to import service "$JS.API.>" on behalf of PUBLIC
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965674 [DBG] 172.19.0.2:7422 - lid:9 - Not permitted to subscribe to "$JS.API.>" on behalf of PUBLIC
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965677 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $LDS.8mRek9GPVzK5CbybhoOS4b]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965679 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.$OBJ.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965681 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.STREAM.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965683 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.$KV.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965685 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.INFO]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965686 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.CONSUMER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965688 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.META.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965690 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.SERVER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965691 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965693 [TRC] 172.19.0.2:7422 - lid:9 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965706 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.API.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965719 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965724 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.$OBJ.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965743 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.INFO]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965747 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965750 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.SERVER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965753 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.$KV.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965755 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.STREAM.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965758 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.CONSUMER.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965770 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.META.>]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:20.965773 [TRC] 172.19.0.2:7422 - lid:9 - <<- [LS+ $LDS.snwsb54Z6k3qSK5oqg3zU9]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:22.067585 [TRC] 172.19.0.2:7422 - lid:9 - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:22.067603 [TRC] 172.19.0.2:7422 - lid:9 - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:22.067624 [DBG] 172.19.0.2:7422 - lid:9 - Leafnode Ping Timer
dev-nats-leaf-1  | [1] 2022/04/11 14:55:22.067632 [TRC] 172.19.0.2:7422 - lid:9 - ->> [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:22.067742 [TRC] 172.19.0.2:7422 - lid:9 - <<- [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.417361 [DBG] 172.19.0.1:42806 - cid:10 - Client connection created
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.417799 [TRC] 172.19.0.1:42806 - cid:10 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.28","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.417886 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PING]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.417896 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [PONG]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418181 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [SUB _INBOX.Rd7VW1R8dMNEBvhwGCxZvk.*  1]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418200 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- [PUB $JS.API.STREAM.INFO.orders _INBOX.Rd7VW1R8dMNEBvhwGCxZvk.mxldJ5NE 0]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418205 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - <<- MSG_PAYLOAD: [""]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418432 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - ->> [MSG _INBOX.Rd7VW1R8dMNEBvhwGCxZvk.mxldJ5NE 1 647]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418447 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1274]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.418460 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"8mRek9GPVzK5CbybhoOS6B\",\"timestamp\":\"2022-04-11T14:55:27.4183973Z\",\"server\":\"NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX\",\"client\":{\"start\":\"2022-04-11T14:55:27.4173164Z\",\"host\":\"172.19.0.1\",\"id\":10,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.28\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":493200,\"server\":\"NB52VH4YKKOWJP3O65FTECPVKXMQLPWTH5U2XIYU6F52ALHE4JWALRWX\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.orders\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-11T14:55:03.4175439Z\\\",\\\"state\\\":{\\\"messages\\\":1,\\\"bytes\\\":61,\\\"first_seq\\\":1,\\\"first_ts\\\":\\\"2022-04-11T14:55:10.4947619Z\\\",\\\"last_seq\\\":1,\\\"last_ts\\\":\\\"2022-04-11T14:55:10.4947619Z\\\",\\\"num_subjects\\\":1,\\\"consumer_count\\\":0},\\\"domain\\\":\\\"leaf\\\"}\",\"domain\":\"leaf\"}"]
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.420948 [DBG] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - Client connection closed: Client Closed
dev-nats-leaf-1  | [1] 2022/04/11 14:55:27.420974 [TRC] 172.19.0.1:42806 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.28" - <-> [DELSUB 1]
kozlovic commented 2 years ago

@JH7 This is because you are using the same store directory for the 2 servers. For instance, if you were to change leaf_test.conf to this:

jetstream {
  domain: leaf
  store_dir: "/tmp/nats/leaf"
}

it will work fine, even after a restart of the leaf server. Give it a try and I am sure it will work. Closing for now, will re-open if you still have issues.

JH7 commented 2 years ago

@kozlovic Thanks for your reply. Nevertheless, this solution does not work for me. I tried configuring store_dir for the leaf node, for the main server and for both the leaf and the main server (with different directories for each of them). The message does not show up in the jetstream after the leaf restart.

Also note that in our production environment, both main and leaf node use seperate volume mounts.

I'm attaching the log for the case in which i configured the store_dir on the leaf node only.

nats-leaf_1  | [1] 2022/04/12 07:07:41.224572 [INF] Starting nats-server
nats-leaf_1  | [1] 2022/04/12 07:07:41.224598 [INF]   Version:  2.7.4
nats-leaf_1  | [1] 2022/04/12 07:07:41.224599 [INF]   Git:      [a86b84a]
nats-leaf_1  | [1] 2022/04/12 07:07:41.224600 [DBG]   Go build: go1.17.8
nats-leaf_1  | [1] 2022/04/12 07:07:41.224605 [INF]   Name:     NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7
nats-leaf_1  | [1] 2022/04/12 07:07:41.224617 [INF]   Node:     Bi6Tknme
nats-leaf_1  | [1] 2022/04/12 07:07:41.224618 [INF]   ID:       NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7
nats-leaf_1  | [1] 2022/04/12 07:07:41.224620 [WRN] Plaintext passwords detected, use nkeys or bcrypt
nats-leaf_1  | [1] 2022/04/12 07:07:41.224623 [INF] Using configuration file: /etc/nats/nats-server.conf
nats-leaf_1  | [1] 2022/04/12 07:07:41.224630 [DBG] Created system account: "$SYS"
nats-leaf_1  | [1] 2022/04/12 07:07:41.224853 [INF] Starting http monitor on 0.0.0.0:8222
nats-leaf_1  | [1] 2022/04/12 07:07:41.224866 [INF] Starting JetStream
nats-leaf_1  | [1] 2022/04/12 07:07:41.225044 [DBG] JetStream creating dynamic configuration - 1.46 GB memory, 21.23 GB disk
nats-leaf_1  | [1] 2022/04/12 07:07:41.225102 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
nats-leaf_1  | [1] 2022/04/12 07:07:41.225105 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
nats-leaf_1  | [1] 2022/04/12 07:07:41.225106 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
nats-leaf_1  | [1] 2022/04/12 07:07:41.225107 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
nats-leaf_1  | [1] 2022/04/12 07:07:41.225108 [INF] 
nats-leaf_1  | [1] 2022/04/12 07:07:41.225109 [INF]          https://docs.nats.io/jetstream
nats-leaf_1  | [1] 2022/04/12 07:07:41.225110 [INF] 
nats-leaf_1  | [1] 2022/04/12 07:07:41.225111 [INF] ---------------- JETSTREAM ----------------
nats-leaf_1  | [1] 2022/04/12 07:07:41.225113 [INF]   Max Memory:      1.46 GB
nats-leaf_1  | [1] 2022/04/12 07:07:41.225115 [INF]   Max Storage:     21.23 GB
nats-leaf_1  | [1] 2022/04/12 07:07:41.225116 [INF]   Store Directory: "/tmp/nats/leaf/jetstream"
nats-leaf_1  | [1] 2022/04/12 07:07:41.225117 [INF]   Domain:          leaf
nats-leaf_1  | [1] 2022/04/12 07:07:41.225118 [INF] -------------------------------------------
nats-leaf_1  | [1] 2022/04/12 07:07:41.225148 [DBG]   Exports:
nats-leaf_1  | [1] 2022/04/12 07:07:41.225151 [DBG]      $JS.API.>
nats-leaf_1  | [1] 2022/04/12 07:07:41.225266 [DBG] Enabled JetStream for account "CLIENT"
nats-leaf_1  | [1] 2022/04/12 07:07:41.225274 [DBG]   Max Memory:      -1 B
nats-leaf_1  | [1] 2022/04/12 07:07:41.225277 [DBG]   Max Storage:     -1 B
nats-leaf_1  | [1] 2022/04/12 07:07:41.225345 [DBG] JetStream state for account "CLIENT" recovered
nats-leaf_1  | [1] 2022/04/12 07:07:41.225409 [INF] Listening for websocket clients on ws://0.0.0.0:4112
nats-leaf_1  | [1] 2022/04/12 07:07:41.225417 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
nats-leaf_1  | [1] 2022/04/12 07:07:41.225419 [DBG] Get non local IPs for "0.0.0.0"
nats-leaf_1  | [1] 2022/04/12 07:07:41.225545 [DBG]   ip=172.18.0.3
nats-leaf_1  | [1] 2022/04/12 07:07:41.225590 [INF] Listening for client connections on 0.0.0.0:4111
nats-leaf_1  | [1] 2022/04/12 07:07:41.225594 [DBG] Get non local IPs for "0.0.0.0"
nats-leaf_1  | [1] 2022/04/12 07:07:41.225683 [DBG]   ip=172.18.0.3
nats-leaf_1  | [1] 2022/04/12 07:07:41.225828 [INF] Server is ready
nats-leaf_1  | [1] 2022/04/12 07:07:41.229128 [DBG] Trying to connect as leafnode to remote server on "nats.local:7422" (172.18.0.2:7422)
nats-leaf_1  | [1] 2022/04/12 07:07:41.229702 [INF] 172.18.0.2:7422 - lid:6 - Leafnode connection created for account: PUBLIC 
nats-leaf_1  | [1] 2022/04/12 07:07:41.231408 [DBG] 172.18.0.2:7422 - lid:6 - Remote leafnode connect msg sent
nats-leaf_1  | [1] 2022/04/12 07:07:41.232299 [INF] 172.18.0.2:7422 - lid:6 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232325 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.$KV.>" -> $KV.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232329 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232332 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.INFO" -> $JS.API.INFO to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232338 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232388 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232393 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.META.>" -> $JS.API.META.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232397 [INF] 172.18.0.2:7422 - lid:6 - Adding JetStream Domain Mapping "$JS.leaf.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232403 [INF] 172.18.0.2:7422 - lid:6 - Adding deny "$JS.leaf.API.>" for outgoing messages to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:41.232408 [DBG] 172.18.0.2:7422 - lid:6 - Not permitted to import service "$JS.API.>" on behalf of PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:07:41.232419 [DBG] 172.18.0.2:7422 - lid:6 - Not permitted to subscribe to "$JS.API.>" on behalf of PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:07:41.232422 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232424 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232426 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232428 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $LDS.ci2eQFvGxYqRwOCuqF3Kzu]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232429 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232485 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232490 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232492 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232493 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232495 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ $JS.leaf.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232499 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232507 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232510 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232512 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232515 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $LDS.1inedEHe104SWImYI2YSqR]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232518 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232521 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232524 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.API.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232527 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232530 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:41.232533 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LS+ $JS.hub.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:07:42.387188 [TRC] 172.18.0.2:7422 - lid:6 - <<- [PING]
nats-leaf_1  | [1] 2022/04/12 07:07:42.387218 [TRC] 172.18.0.2:7422 - lid:6 - ->> [PONG]
nats-leaf_1  | [1] 2022/04/12 07:07:42.399785 [DBG] 172.18.0.2:7422 - lid:6 - Leafnode Ping Timer
nats-leaf_1  | [1] 2022/04/12 07:07:42.400554 [TRC] 172.18.0.2:7422 - lid:6 - ->> [PING]
nats-leaf_1  | [1] 2022/04/12 07:07:42.402086 [TRC] 172.18.0.2:7422 - lid:6 - <<- [PONG]
nats-leaf_1  | [1] 2022/04/12 07:07:45.695301 [DBG] 172.18.0.1:60162 - cid:7 - Client connection created
nats-leaf_1  | [1] 2022/04/12 07:07:45.696858 [TRC] 172.18.0.1:60162 - cid:7 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.30","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
nats-leaf_1  | [1] 2022/04/12 07:07:45.697160 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-leaf_1  | [1] 2022/04/12 07:07:45.697175 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-leaf_1  | [1] 2022/04/12 07:07:45.699839 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [SUB _INBOX.irrOFD2xZhmkosWbT7yK78.*  1]
nats-leaf_1  | [1] 2022/04/12 07:07:45.699888 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PUB $JS.API.STREAM.CREATE.orders _INBOX.irrOFD2xZhmkosWbT7yK78.LarxNJHR 344]
nats-leaf_1  | [1] 2022/04/12 07:07:45.700068 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- MSG_PAYLOAD: ["{\"name\":\"orders\",\"subjects\":[\"orders.*.stream.entry\"],\"retention\":\"workqueue\",\"max_consumers\":-1,\"max_msgs_per_subject\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msg_size\":-1,\"storage\":\"file\",\"discard\":\"old\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"sealed\":false,\"deny_delete\":false,\"deny_purge\":false,\"allow_rollup_hdrs\":false}"]
nats-leaf_1  | [1] 2022/04/12 07:07:45.701593 [DBG] JETSTREAM - Creating import subscription on "orders.*.stream.entry" from account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:45.701633 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS+ orders.*.stream.entry]
nats-leaf_1  | [1] 2022/04/12 07:07:45.702256 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [MSG _INBOX.irrOFD2xZhmkosWbT7yK78.LarxNJHR 1 619]
nats-leaf_1  | [1] 2022/04/12 07:07:45.702278 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1649]
nats-leaf_1  | [1] 2022/04/12 07:07:45.702346 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"ci2eQFvGxYqRwOCuqF3L4V\",\"timestamp\":\"2022-04-12T07:07:45.702132962Z\",\"server\":\"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7\",\"client\":{\"start\":\"2022-04-12T07:07:45.695122878Z\",\"host\":\"172.18.0.1\",\"id\":7,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.30\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":1757292,\"server\":\"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.CREATE.orders\",\"request\":\"{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs_per_subject\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"discard\\\":\\\"old\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_create_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-12T07:07:45.700574087Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"did_create\\\":true}\",\"domain\":\"leaf\"}"]
nats-leaf_1  | [1] 2022/04/12 07:07:45.707179 [DBG] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - Client connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:45.707222 [TRC] 172.18.0.1:60162 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <-> [DELSUB 1]
nats-leaf_1  | [1] 2022/04/12 07:07:48.396502 [TRC] 172.18.0.2:7422 - lid:6 - <<- [LMSG orders.client.stream.entry 5]
nats-leaf_1  | [1] 2022/04/12 07:07:48.396545 [TRC] 172.18.0.2:7422 - lid:6 - <<- MSG_PAYLOAD: ["test1"]
nats-leaf_1  | [1] 2022/04/12 07:07:50.469064 [DBG] 172.18.0.1:60166 - cid:12 - Client connection created
nats-leaf_1  | [1] 2022/04/12 07:07:50.470686 [TRC] 172.18.0.1:60166 - cid:12 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.30","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
nats-leaf_1  | [1] 2022/04/12 07:07:50.470796 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-leaf_1  | [1] 2022/04/12 07:07:50.470803 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-leaf_1  | [1] 2022/04/12 07:07:50.473191 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [SUB _INBOX.gz596PPHmMTQMZAGELwd1D.*  1]
nats-leaf_1  | [1] 2022/04/12 07:07:50.473267 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PUB $JS.API.STREAM.INFO.orders _INBOX.gz596PPHmMTQMZAGELwd1D.kGwrmcZM 4]
nats-leaf_1  | [1] 2022/04/12 07:07:50.473303 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- MSG_PAYLOAD: ["null"]
nats-leaf_1  | [1] 2022/04/12 07:07:50.474143 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [MSG _INBOX.gz596PPHmMTQMZAGELwd1D.kGwrmcZM 1 653]
nats-leaf_1  | [1] 2022/04/12 07:07:50.474235 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1302]
nats-leaf_1  | [1] 2022/04/12 07:07:50.474330 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"ci2eQFvGxYqRwOCuqF3L62\",\"timestamp\":\"2022-04-12T07:07:50.473756006Z\",\"server\":\"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7\",\"client\":{\"start\":\"2022-04-12T07:07:50.468974464Z\",\"host\":\"172.18.0.1\",\"id\":12,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.30\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":1737542,\"server\":\"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.orders\",\"request\":\"null\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-12T07:07:45.700574087Z\\\",\\\"state\\\":{\\\"messages\\\":1,\\\"bytes\\\":61,\\\"first_seq\\\":1,\\\"first_ts\\\":\\\"2022-04-12T07:07:48.396603005Z\\\",\\\"last_seq\\\":1,\\\"last_ts\\\":\\\"2022-04-12T07:07:48.396603005Z\\\",\\\"num_subjects\\\":1,\\\"consumer_count\\\":0},\\\"domain\\\":\\\"leaf\\\"}\",\"domain\":\"leaf\"}"]
nats-leaf_1  | [1] 2022/04/12 07:07:50.479669 [DBG] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - Client connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:50.479759 [TRC] 172.18.0.1:60166 - cid:12 - "v1.13.0:go:NATS CLI Version 0.0.30" - <-> [DELSUB 1]
nats-leaf_1  | [1] 2022/04/12 07:07:54.272998 [DBG] Trapped "terminated" signal
nats-leaf_1  | [1] 2022/04/12 07:07:54.273076 [INF] Initiating Shutdown...
nats-leaf_1  | [1] 2022/04/12 07:07:54.273096 [INF] Initiating JetStream Shutdown...
nats-leaf_1  | [1] 2022/04/12 07:07:54.273123 [TRC] JETSTREAM - <-> [DELSUB 1]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273146 [TRC] 172.18.0.2:7422 - lid:6 - ->> [LS- orders.*.stream.entry]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273153 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:54.273157 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:54.273240 [INF] JetStream Shutdown
nats-leaf_1  | [1] 2022/04/12 07:07:54.273326 [INF] 172.18.0.2:7422 - lid:6 - Leafnode connection closed: Server Shutdown account: PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:07:54.273341 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273345 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.API.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273349 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273352 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273355 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273358 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273360 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273363 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273365 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273368 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $LDS.1inedEHe104SWImYI2YSqR]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273370 [TRC] 172.18.0.2:7422 - lid:6 - <-> [DELSUB $JS.hub.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:07:54.273507 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:54.273516 [DBG] Client accept loop exiting..
nats-leaf_1  | [1] 2022/04/12 07:07:54.273528 [DBG] SYSTEM - System connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:07:54.273535 [INF] Server Exiting..
nats-leaf_1  | [1] 2022/04/12 07:07:59.966288 [INF] Starting nats-server
nats-leaf_1  | [1] 2022/04/12 07:07:59.966315 [INF]   Version:  2.7.4
nats-leaf_1  | [1] 2022/04/12 07:07:59.966317 [INF]   Git:      [a86b84a]
nats-leaf_1  | [1] 2022/04/12 07:07:59.966318 [DBG]   Go build: go1.17.8
nats-leaf_1  | [1] 2022/04/12 07:07:59.966319 [INF]   Name:     ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5
nats-leaf_1  | [1] 2022/04/12 07:07:59.966324 [INF]   Node:     lcv1AG3B
nats-leaf_1  | [1] 2022/04/12 07:07:59.966325 [INF]   ID:       ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5
nats-leaf_1  | [1] 2022/04/12 07:07:59.966327 [WRN] Plaintext passwords detected, use nkeys or bcrypt
nats-leaf_1  | [1] 2022/04/12 07:07:59.966334 [INF] Using configuration file: /etc/nats/nats-server.conf
nats-leaf_1  | [1] 2022/04/12 07:07:59.966341 [DBG] Created system account: "$SYS"
nats-leaf_1  | [1] 2022/04/12 07:07:59.966569 [INF] Starting http monitor on 0.0.0.0:8222
nats-leaf_1  | [1] 2022/04/12 07:07:59.966594 [INF] Starting JetStream
nats-leaf_1  | [1] 2022/04/12 07:07:59.966631 [DBG] JetStream creating dynamic configuration - 1.46 GB memory, 21.23 GB disk
nats-leaf_1  | [1] 2022/04/12 07:07:59.966694 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
nats-leaf_1  | [1] 2022/04/12 07:07:59.966696 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
nats-leaf_1  | [1] 2022/04/12 07:07:59.966697 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
nats-leaf_1  | [1] 2022/04/12 07:07:59.966697 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
nats-leaf_1  | [1] 2022/04/12 07:07:59.966698 [INF] 
nats-leaf_1  | [1] 2022/04/12 07:07:59.966699 [INF]          https://docs.nats.io/jetstream
nats-leaf_1  | [1] 2022/04/12 07:07:59.966700 [INF] 
nats-leaf_1  | [1] 2022/04/12 07:07:59.966701 [INF] ---------------- JETSTREAM ----------------
nats-leaf_1  | [1] 2022/04/12 07:07:59.966703 [INF]   Max Memory:      1.46 GB
nats-leaf_1  | [1] 2022/04/12 07:07:59.966704 [INF]   Max Storage:     21.23 GB
nats-leaf_1  | [1] 2022/04/12 07:07:59.966706 [INF]   Store Directory: "/tmp/nats/leaf/jetstream"
nats-leaf_1  | [1] 2022/04/12 07:07:59.966706 [INF]   Domain:          leaf
nats-leaf_1  | [1] 2022/04/12 07:07:59.966707 [INF] -------------------------------------------
nats-leaf_1  | [1] 2022/04/12 07:07:59.966739 [DBG]   Exports:
nats-leaf_1  | [1] 2022/04/12 07:07:59.966740 [DBG]      $JS.API.>
nats-leaf_1  | [1] 2022/04/12 07:07:59.966820 [DBG] Enabled JetStream for account "CLIENT"
nats-leaf_1  | [1] 2022/04/12 07:07:59.966823 [DBG]   Max Memory:      -1 B
nats-leaf_1  | [1] 2022/04/12 07:07:59.966824 [DBG]   Max Storage:     -1 B
nats-leaf_1  | [1] 2022/04/12 07:07:59.966839 [DBG] Recovering JetStream state for account "CLIENT"
nats-leaf_1  | [1] 2022/04/12 07:07:59.968984 [DBG] JETSTREAM - Creating import subscription on "orders.*.stream.entry" from account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.969031 [INF]   Restored 1 messages for stream 'CLIENT > orders'
nats-leaf_1  | [1] 2022/04/12 07:07:59.969245 [DBG] JetStream state for account "CLIENT" recovered
nats-leaf_1  | [1] 2022/04/12 07:07:59.969340 [INF] Listening for websocket clients on ws://0.0.0.0:4112
nats-leaf_1  | [1] 2022/04/12 07:07:59.969342 [WRN] Websocket not configured with TLS. DO NOT USE IN PRODUCTION!
nats-leaf_1  | [1] 2022/04/12 07:07:59.969344 [DBG] Get non local IPs for "0.0.0.0"
nats-leaf_1  | [1] 2022/04/12 07:07:59.969462 [DBG]   ip=172.18.0.3
nats-leaf_1  | [1] 2022/04/12 07:07:59.969483 [INF] Listening for client connections on 0.0.0.0:4111
nats-leaf_1  | [1] 2022/04/12 07:07:59.969485 [DBG] Get non local IPs for "0.0.0.0"
nats-leaf_1  | [1] 2022/04/12 07:07:59.969584 [DBG]   ip=172.18.0.3
nats-leaf_1  | [1] 2022/04/12 07:07:59.969589 [INF] Server is ready
nats-leaf_1  | [1] 2022/04/12 07:07:59.970327 [DBG] Trying to connect as leafnode to remote server on "nats.local:7422" (172.18.0.2:7422)
nats-leaf_1  | [1] 2022/04/12 07:07:59.970672 [INF] 172.18.0.2:7422 - lid:9 - Leafnode connection created for account: PUBLIC 
nats-leaf_1  | [1] 2022/04/12 07:07:59.970840 [DBG] 172.18.0.2:7422 - lid:9 - Remote leafnode connect msg sent
nats-leaf_1  | [1] 2022/04/12 07:07:59.971714 [INF] 172.18.0.2:7422 - lid:9 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971739 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.INFO" -> $JS.API.INFO to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971757 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971764 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971767 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.META.>" -> $JS.API.META.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971771 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971774 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.$KV.>" -> $KV.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971778 [INF] 172.18.0.2:7422 - lid:9 - Adding JetStream Domain Mapping "$JS.leaf.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971781 [INF] 172.18.0.2:7422 - lid:9 - Adding deny "$JS.leaf.API.>" for outgoing messages to account "PUBLIC"
nats-leaf_1  | [1] 2022/04/12 07:07:59.971787 [DBG] 172.18.0.2:7422 - lid:9 - Not permitted to import service "$JS.API.>" on behalf of PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:07:59.971793 [DBG] 172.18.0.2:7422 - lid:9 - Not permitted to subscribe to "$JS.API.>" on behalf of PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:07:59.971797 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971799 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971800 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971802 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $LDS.3N1KeTWqtLgyn3luTFEsgO]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971804 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971805 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971809 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971810 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971812 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971813 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS+ $JS.leaf.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971821 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971826 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.API.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971828 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971831 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971834 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971836 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971839 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971849 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971860 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971863 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $JS.hub.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:07:59.971867 [TRC] 172.18.0.2:7422 - lid:9 - <<- [LS+ $LDS.1inedEHe104SWImYI2YSqR]
nats-leaf_1  | [1] 2022/04/12 07:08:01.110880 [TRC] 172.18.0.2:7422 - lid:9 - <<- [PING]
nats-leaf_1  | [1] 2022/04/12 07:08:01.111144 [TRC] 172.18.0.2:7422 - lid:9 - ->> [PONG]
nats-leaf_1  | [1] 2022/04/12 07:08:01.119855 [DBG] 172.18.0.2:7422 - lid:9 - Leafnode Ping Timer
nats-leaf_1  | [1] 2022/04/12 07:08:01.120154 [TRC] 172.18.0.2:7422 - lid:9 - ->> [PING]
nats-leaf_1  | [1] 2022/04/12 07:08:01.121214 [TRC] 172.18.0.2:7422 - lid:9 - <<- [PONG]
nats-leaf_1  | [1] 2022/04/12 07:08:06.162161 [DBG] 172.18.0.1:60172 - cid:10 - Client connection created
nats-leaf_1  | [1] 2022/04/12 07:08:06.163879 [TRC] 172.18.0.1:60172 - cid:10 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"client","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.30","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
nats-leaf_1  | [1] 2022/04/12 07:08:06.164162 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-leaf_1  | [1] 2022/04/12 07:08:06.164172 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-leaf_1  | [1] 2022/04/12 07:08:06.165578 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [SUB _INBOX.U5OpVsfD82kA0arrPMaOuT.*  1]
nats-leaf_1  | [1] 2022/04/12 07:08:06.165627 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PUB $JS.API.STREAM.INFO.orders _INBOX.U5OpVsfD82kA0arrPMaOuT.rjyMyX7E 4]
nats-leaf_1  | [1] 2022/04/12 07:08:06.165639 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- MSG_PAYLOAD: ["null"]
nats-leaf_1  | [1] 2022/04/12 07:08:06.166690 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [MSG _INBOX.U5OpVsfD82kA0arrPMaOuT.rjyMyX7E 1 653]
nats-leaf_1  | [1] 2022/04/12 07:08:06.166751 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1302]
nats-leaf_1  | [1] 2022/04/12 07:08:06.166842 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"3N1KeTWqtLgyn3luTFEspA\",\"timestamp\":\"2022-04-12T07:08:06.166481763Z\",\"server\":\"ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5\",\"client\":{\"start\":\"2022-04-12T07:08:06.161887472Z\",\"host\":\"172.18.0.1\",\"id\":10,\"acc\":\"CLIENT\",\"user\":\"client\",\"name\":\"NATS CLI Version 0.0.30\",\"lang\":\"go\",\"ver\":\"1.13.0\",\"rtt\":2017250,\"server\":\"ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.orders\",\"request\":\"null\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"config\\\":{\\\"name\\\":\\\"orders\\\",\\\"subjects\\\":[\\\"orders.*.stream.entry\\\"],\\\"retention\\\":\\\"workqueue\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":-1,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"sealed\\\":false,\\\"deny_delete\\\":false,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":false},\\\"created\\\":\\\"2022-04-12T07:07:45.700574087Z\\\",\\\"state\\\":{\\\"messages\\\":1,\\\"bytes\\\":61,\\\"first_seq\\\":1,\\\"first_ts\\\":\\\"2022-04-12T07:07:48.396603005Z\\\",\\\"last_seq\\\":1,\\\"last_ts\\\":\\\"2022-04-12T07:07:48.396603005Z\\\",\\\"num_subjects\\\":1,\\\"consumer_count\\\":0},\\\"domain\\\":\\\"leaf\\\"}\",\"domain\":\"leaf\"}"]
nats-leaf_1  | [1] 2022/04/12 07:08:06.173346 [DBG] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - Client connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:08:06.173474 [TRC] 172.18.0.1:60172 - cid:10 - "v1.13.0:go:NATS CLI Version 0.0.30" - <-> [DELSUB 1]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595246 [DBG] Trapped "terminated" signal
nats-leaf_1  | [1] 2022/04/12 07:08:13.595435 [INF] Initiating Shutdown...
nats-leaf_1  | [1] 2022/04/12 07:08:13.595458 [INF] Initiating JetStream Shutdown...
nats-leaf_1  | [1] 2022/04/12 07:08:13.595484 [TRC] JETSTREAM - <-> [DELSUB 1]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595507 [TRC] 172.18.0.2:7422 - lid:9 - ->> [LS- orders.*.stream.entry]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595520 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:08:13.595523 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:08:13.595610 [DBG] JETSTREAM - JetStream connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:08:13.595611 [INF] JetStream Shutdown
nats-leaf_1  | [1] 2022/04/12 07:08:13.595671 [INF] 172.18.0.2:7422 - lid:9 - Leafnode connection closed: Server Shutdown account: PUBLIC
nats-leaf_1  | [1] 2022/04/12 07:08:13.595683 [DBG] Client accept loop exiting..
nats-leaf_1  | [1] 2022/04/12 07:08:13.595693 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595695 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.$OBJ.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595697 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.CONSUMER.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595701 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595703 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.API.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595706 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.META.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595707 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.INFO]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595709 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.STREAM.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595711 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $LDS.1inedEHe104SWImYI2YSqR]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595713 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.SERVER.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595715 [TRC] 172.18.0.2:7422 - lid:9 - <-> [DELSUB $JS.hub.API.$KV.>]
nats-leaf_1  | [1] 2022/04/12 07:08:13.595729 [DBG] SYSTEM - System connection closed: Client Closed
nats-leaf_1  | [1] 2022/04/12 07:08:13.596035 [INF] Server Exiting..

nats-base_1  | [1] 2022/04/12 07:07:40.821492 [INF] Starting nats-server
nats-base_1  | [1] 2022/04/12 07:07:40.821518 [INF]   Version:  2.7.4
nats-base_1  | [1] 2022/04/12 07:07:40.821519 [INF]   Git:      [a86b84a]
nats-base_1  | [1] 2022/04/12 07:07:40.821521 [DBG]   Go build: go1.17.8
nats-base_1  | [1] 2022/04/12 07:07:40.821522 [INF]   Name:     NATYQI7OBDQIPT7MJQBPLHREMSE7AM7FZATH2LTY2Q4LOCHCU4OSNAC4
nats-base_1  | [1] 2022/04/12 07:07:40.821524 [INF]   Node:     Hv3yGllV
nats-base_1  | [1] 2022/04/12 07:07:40.821525 [INF]   ID:       NATYQI7OBDQIPT7MJQBPLHREMSE7AM7FZATH2LTY2Q4LOCHCU4OSNAC4
nats-base_1  | [1] 2022/04/12 07:07:40.821527 [WRN] Plaintext passwords detected, use nkeys or bcrypt
nats-base_1  | [1] 2022/04/12 07:07:40.821532 [INF] Using configuration file: /etc/nats/nats-server.conf
nats-base_1  | [1] 2022/04/12 07:07:40.821539 [DBG] Created system account: "$SYS"
nats-base_1  | [1] 2022/04/12 07:07:40.822117 [INF] Starting http monitor on 0.0.0.0:8222
nats-base_1  | [1] 2022/04/12 07:07:40.822141 [INF] Starting JetStream
nats-base_1  | [1] 2022/04/12 07:07:40.822332 [DBG] JetStream creating dynamic configuration - 1.46 GB memory, 21.23 GB disk
nats-base_1  | [1] 2022/04/12 07:07:40.822450 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
nats-base_1  | [1] 2022/04/12 07:07:40.822454 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
nats-base_1  | [1] 2022/04/12 07:07:40.822455 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
nats-base_1  | [1] 2022/04/12 07:07:40.822456 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
nats-base_1  | [1] 2022/04/12 07:07:40.822457 [INF] 
nats-base_1  | [1] 2022/04/12 07:07:40.822458 [INF]          https://docs.nats.io/jetstream
nats-base_1  | [1] 2022/04/12 07:07:40.822459 [INF] 
nats-base_1  | [1] 2022/04/12 07:07:40.822460 [INF] ---------------- JETSTREAM ----------------
nats-base_1  | [1] 2022/04/12 07:07:40.822477 [INF]   Max Memory:      1.46 GB
nats-base_1  | [1] 2022/04/12 07:07:40.822480 [INF]   Max Storage:     21.23 GB
nats-base_1  | [1] 2022/04/12 07:07:40.822481 [INF]   Store Directory: "/tmp/nats/jetstream"
nats-base_1  | [1] 2022/04/12 07:07:40.822482 [INF]   Domain:          hub
nats-base_1  | [1] 2022/04/12 07:07:40.822483 [INF] -------------------------------------------
nats-base_1  | [1] 2022/04/12 07:07:40.822512 [DBG]   Exports:
nats-base_1  | [1] 2022/04/12 07:07:40.822514 [DBG]      $JS.API.>
nats-base_1  | [1] 2022/04/12 07:07:40.822597 [DBG] Enabled JetStream for account "DMZ"
nats-base_1  | [1] 2022/04/12 07:07:40.822600 [DBG]   Max Memory:      -1 B
nats-base_1  | [1] 2022/04/12 07:07:40.822601 [DBG]   Max Storage:     -1 B
nats-base_1  | [1] 2022/04/12 07:07:40.822695 [DBG] JetStream state for account "DMZ" recovered
nats-base_1  | [1] 2022/04/12 07:07:40.822754 [INF] Listening for leafnode connections on 0.0.0.0:7422
nats-base_1  | [1] 2022/04/12 07:07:40.822756 [DBG] Get non local IPs for "0.0.0.0"
nats-base_1  | [1] 2022/04/12 07:07:40.822858 [DBG]   ip=172.18.0.2
nats-base_1  | [1] 2022/04/12 07:07:40.823008 [INF] Listening for client connections on 0.0.0.0:4222
nats-base_1  | [1] 2022/04/12 07:07:40.823011 [DBG] Get non local IPs for "0.0.0.0"
nats-base_1  | [1] 2022/04/12 07:07:40.823104 [DBG]   ip=172.18.0.2
nats-base_1  | [1] 2022/04/12 07:07:40.823114 [INF] Server is ready
nats-base_1  | [1] 2022/04/12 07:07:41.230583 [INF] 172.18.0.3:50524 - lid:6 - Leafnode connection created 
nats-base_1  | [1] 2022/04/12 07:07:41.231808 [TRC] 172.18.0.3:50524 - lid:6 - <<- [CONNECT {"user":"leaf","pass":"[REDACTED]","tls_required":false,"server_id":"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7","domain":"leaf","name":"NBWLD2W77VGB2SKMNJ6BALGWCM5NVXOE2SMUSIBKHPR2EJQ6LSLHGQR7","headers":true}]
nats-base_1  | [1] 2022/04/12 07:07:41.232080 [INF] 172.18.0.3:50524 - lid:6 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232110 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232114 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.META.>" -> $JS.API.META.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232118 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232121 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.$KV.>" -> $KV.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232124 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232127 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.INFO" -> $JS.API.INFO to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232130 [INF] 172.18.0.3:50524 - lid:6 - Adding JetStream Domain Mapping "$JS.hub.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232135 [INF] 172.18.0.3:50524 - lid:6 - Adding deny "$JS.hub.API.>" for outgoing messages to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:41.232184 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232186 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232188 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232190 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232191 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $LDS.1inedEHe104SWImYI2YSqR]
nats-base_1  | [1] 2022/04/12 07:07:41.232194 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:41.232195 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:41.232197 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.API.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232199 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.INFO]
nats-base_1  | [1] 2022/04/12 07:07:41.232200 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232202 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LS+ $JS.hub.API.META.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232598 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:41.232605 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232609 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232611 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $LDS.ci2eQFvGxYqRwOCuqF3Kzu]
nats-base_1  | [1] 2022/04/12 07:07:41.232616 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:41.232619 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.INFO]
nats-base_1  | [1] 2022/04/12 07:07:41.232622 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232624 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.META.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232627 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:07:41.232631 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ $JS.leaf.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:07:42.385857 [DBG] 172.18.0.3:50524 - lid:6 - Leafnode Ping Timer
nats-base_1  | [1] 2022/04/12 07:07:42.386432 [TRC] 172.18.0.3:50524 - lid:6 - ->> [PING]
nats-base_1  | [1] 2022/04/12 07:07:42.387558 [TRC] 172.18.0.3:50524 - lid:6 - <<- [PONG]
nats-base_1  | [1] 2022/04/12 07:07:42.401278 [TRC] 172.18.0.3:50524 - lid:6 - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:07:42.401322 [TRC] 172.18.0.3:50524 - lid:6 - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:07:45.702335 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS+ orders.*.stream.entry]
nats-base_1  | [1] 2022/04/12 07:07:45.702390 [DBG] 172.18.0.3:50524 - lid:6 - Creating import subscription on "orders.*.stream.entry" from account "DMZ"
nats-base_1  | [1] 2022/04/12 07:07:48.389352 [DBG] 172.18.0.1:64424 - cid:7 - Client connection created
nats-base_1  | [1] 2022/04/12 07:07:48.392328 [TRC] 172.18.0.1:64424 - cid:7 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"service","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.30","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
nats-base_1  | [1] 2022/04/12 07:07:48.392557 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:07:48.392570 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:07:48.395556 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PUB orders.client.stream.entry 5]
nats-base_1  | [1] 2022/04/12 07:07:48.395615 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- MSG_PAYLOAD: ["test1"]
nats-base_1  | [1] 2022/04/12 07:07:48.395687 [TRC] 172.18.0.3:50524 - lid:6 - ->> [LMSG orders.client.stream.entry 5]
nats-base_1  | [1] 2022/04/12 07:07:48.395697 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:07:48.395705 [TRC] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:07:48.398091 [DBG] 172.18.0.1:64424 - cid:7 - "v1.13.0:go:NATS CLI Version 0.0.30" - Client connection closed: Client Closed
nats-base_1  | [1] 2022/04/12 07:07:54.273530 [TRC] 172.18.0.3:50524 - lid:6 - <<- [LS- orders.*.stream.entry]
nats-base_1  | [1] 2022/04/12 07:07:54.273545 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB orders.*.stream.entry]
nats-base_1  | [1] 2022/04/12 07:07:54.273561 [INF] 172.18.0.3:50524 - lid:6 - Leafnode connection closed: Client Closed account: PUBLIC
nats-base_1  | [1] 2022/04/12 07:07:54.273578 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $LDS.ci2eQFvGxYqRwOCuqF3Kzu]
nats-base_1  | [1] 2022/04/12 07:07:54.273583 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273586 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273589 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.META.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273591 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273593 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:54.273595 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273598 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:07:54.273601 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:54.273604 [TRC] 172.18.0.3:50524 - lid:6 - <-> [DELSUB $JS.leaf.API.INFO]
nats-base_1  | [1] 2022/04/12 07:07:59.970555 [INF] 172.18.0.3:50532 - lid:8 - Leafnode connection created 
nats-base_1  | [1] 2022/04/12 07:07:59.971467 [TRC] 172.18.0.3:50532 - lid:8 - <<- [CONNECT {"user":"leaf","pass":"[REDACTED]","tls_required":false,"server_id":"ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5","domain":"leaf","name":"ND4D73GUIDHVNAFFBSKIWOOY7SP4ZGD53MJCV6AZKYUZ65UZKPJHCUF5","headers":true}]
nats-base_1  | [1] 2022/04/12 07:07:59.971521 [INF] 172.18.0.3:50532 - lid:8 - JetStream Not Extended, adding deny [$JS.API.> $KV.> $OBJ.>] for account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971548 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.$OBJ.>" -> $OBJ.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971551 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.INFO" -> $JS.API.INFO to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971555 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.STREAM.>" -> $JS.API.STREAM.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971560 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.CONSUMER.>" -> $JS.API.CONSUMER.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971563 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.META.>" -> $JS.API.META.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971566 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.SERVER.>" -> $JS.API.SERVER.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971569 [INF] 172.18.0.3:50532 - lid:8 - Adding JetStream Domain Mapping "$JS.hub.API.$KV.>" -> $KV.> to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971577 [INF] 172.18.0.3:50532 - lid:8 - Adding deny "$JS.hub.API.>" for outgoing messages to account "PUBLIC"
nats-base_1  | [1] 2022/04/12 07:07:59.971603 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:59.971605 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.API.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971608 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971610 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971611 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.META.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971613 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:59.971615 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971617 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.INFO]
nats-base_1  | [1] 2022/04/12 07:07:59.971618 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971620 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $JS.hub.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:07:59.971621 [TRC] 172.18.0.3:50532 - lid:8 - ->> [LS+ $LDS.1inedEHe104SWImYI2YSqR]
nats-base_1  | [1] 2022/04/12 07:07:59.972040 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:07:59.972053 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.INFO]
nats-base_1  | [1] 2022/04/12 07:07:59.972059 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:07:59.972067 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $LDS.3N1KeTWqtLgyn3luTFEsgO]
nats-base_1  | [1] 2022/04/12 07:07:59.972070 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:07:59.972073 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:07:59.972079 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:59.972083 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:07:59.972086 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.META.>]
nats-base_1  | [1] 2022/04/12 07:07:59.972089 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS+ $JS.leaf.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:08:01.109709 [DBG] 172.18.0.3:50532 - lid:8 - Leafnode Ping Timer
nats-base_1  | [1] 2022/04/12 07:08:01.109871 [TRC] 172.18.0.3:50532 - lid:8 - ->> [PING]
nats-base_1  | [1] 2022/04/12 07:08:01.111782 [TRC] 172.18.0.3:50532 - lid:8 - <<- [PONG]
nats-base_1  | [1] 2022/04/12 07:08:01.120650 [TRC] 172.18.0.3:50532 - lid:8 - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:08:01.120734 [TRC] 172.18.0.3:50532 - lid:8 - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:08:03.799730 [DBG] 172.18.0.1:64430 - cid:9 - Client connection created
nats-base_1  | [1] 2022/04/12 07:08:03.801633 [TRC] 172.18.0.1:64430 - cid:9 - <<- [CONNECT {"verbose":false,"pedantic":false,"user":"service","pass":"[REDACTED]","tls_required":false,"name":"NATS CLI Version 0.0.30","lang":"go","version":"1.13.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
nats-base_1  | [1] 2022/04/12 07:08:03.801745 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:08:03.801751 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:08:03.804194 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PUB orders.client.stream.entry 5]
nats-base_1  | [1] 2022/04/12 07:08:03.804260 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- MSG_PAYLOAD: ["test2"]
nats-base_1  | [1] 2022/04/12 07:08:03.804280 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - <<- [PING]
nats-base_1  | [1] 2022/04/12 07:08:03.804289 [TRC] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - ->> [PONG]
nats-base_1  | [1] 2022/04/12 07:08:03.807649 [DBG] 172.18.0.1:64430 - cid:9 - "v1.13.0:go:NATS CLI Version 0.0.30" - Client connection closed: Client Closed
nats-base_1  | [1] 2022/04/12 07:08:13.595808 [TRC] 172.18.0.3:50532 - lid:8 - <<- [LS- orders.*.stream.entry]
nats-base_1  | [1] 2022/04/12 07:08:13.595871 [INF] 172.18.0.3:50532 - lid:8 - Leafnode connection closed: Client Closed account: PUBLIC
nats-base_1  | [1] 2022/04/12 07:08:13.595887 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.$KV.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595889 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.SERVER.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595892 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.INFO]
nats-base_1  | [1] 2022/04/12 07:08:13.595893 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.$OBJ.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595895 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $SYS.REQ.ACCOUNT.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:08:13.595897 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.META.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595917 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.CONSUMER.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595922 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $LDS.3N1KeTWqtLgyn3luTFEsgO]
nats-base_1  | [1] 2022/04/12 07:08:13.595924 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $JS.leaf.API.STREAM.>]
nats-base_1  | [1] 2022/04/12 07:08:13.595926 [TRC] 172.18.0.3:50532 - lid:8 - <-> [DELSUB $SYS.REQ.SERVER.PING.CONNZ]
nats-base_1  | [1] 2022/04/12 07:08:13.705589 [DBG] Trapped "terminated" signal
nats-base_1  | [1] 2022/04/12 07:08:13.705644 [INF] Initiating Shutdown...
nats-base_1  | [1] 2022/04/12 07:08:13.705653 [INF] Initiating JetStream Shutdown...
nats-base_1  | [1] 2022/04/12 07:08:13.705656 [INF] JetStream Shutdown
nats-base_1  | [1] 2022/04/12 07:08:13.705710 [DBG] SYSTEM - System connection closed: Client Closed
nats-base_1  | [1] 2022/04/12 07:08:13.705713 [DBG] Client accept loop exiting..
nats-base_1  | [1] 2022/04/12 07:08:13.705716 [DBG] Leafnode accept loop exiting..
nats-base_1  | [1] 2022/04/12 07:08:13.705720 [INF] Server Exiting..
derekcollison commented 2 years ago

You need to use a persistent server_name that survives restarts.

kozlovic commented 2 years ago

@JH7 The issue should have been resolved now. Thank you for the report.

JH7 commented 2 years ago

@kozlovic thank you for the quick fix!