NATS: Fix illegal resource version from storage: 0 (#274)

jrovira-kumori commented 4 months ago

Fix for #274.

I am guessing this issue did not appear with K3s because the first command it runs when initializing is a CREATE /bootstrap/.... Therefore future COUNT ... calls would always return != 0.

This is not the case with Kubernetes KubeAPI which reads before writing anything and gets stuck in a loop of illegal resource version from storage: 0.

This PR fixes the issue with the NATS driver by incrementing the BucketRevision number to 1 by creating and deleting a bootstrap key if a BucketRevision of 0 is detected at startup.

brandond commented 4 months ago

Thanks for the contribution! Please sign-off your commit for DCO.

@bruth can you comment if this is the best way to do this?

jrovira-kumori commented 4 months ago

Thanks for the contribution! Please sign-off your commit for DCO.

@bruth can you comment if this is the best way to do this?

Done!

brandond commented 4 months ago

fwiw the bootstrap key is internal to k3s, I'm not sure this is the correct thing to create in kine.

nats should already create a /registry/health key on startup. Do we just need another second write to that key to increment the revision to 1? https://github.com/k3s-io/kine/blob/37736729c40f1e7ad18521d1eb5c2a13e314ebde/pkg/drivers/nats/backend.go#L116-L125

jrovira-kumori commented 4 months ago

The /registry/health is never added in the minimal repro or other tests I have done. It has only worked whenever I was debugging the Kine process and added a breakpoint.

There seems to be some kind of race condition going on which causes the error Failed to create health check key: context deadline exceeded to happen.

Here is an example of the Kine logs from the repro.

level=debug msg="using config &nats.Config{clientURL:\"nats://nats:4222\", clientOptions:[]nats.Option(nil), revHistory:0xa, bucket:\"kine\", replicas:1, slowThreshold:500000000, noEmbed:true, dontListen:false, serverConfig:\"\", stdoutLogging:false, host:\"nats\", port:4222, dataDir:\"\"}"
level=info msg="connecting to nats://nats:4222"
level=info msg="using bucket: kine"
level=info msg="metrics server is starting to listen at :8080"
level=info msg="starting metrics server path /metrics"
level=info msg="bucket initialized: kine"
level=error msg="Failed to create health check key: context deadline exceeded"
level=info msg="Kine available at http://127.0.0.1:2379"
level=trace msg="LIST /registry/apiextensions.k8s.io/customresourcedefinitions/, start=/registry/apiextensions.k8s.io/customresourcedefinitions/, limit=10001, rev=0 => rev=0, kvs=0, err=<nil>, duration=2.334µs"
level=trace msg="LIST key=/registry/apiextensions.k8s.io/customresourcedefinitions/, end=/registry/apiextensions.k8s.io/customresourcedefinitions/, revision=0, currentRev=0 count=0, limit=10000"
level=trace msg="COUNT /registry/apiextensions.k8s.io/customresourcedefinitions/, rev=0 => rev=0, count=0, err=<nil>, duration=461ns"
level=trace msg="LIST COUNT key=/registry/apiextensions.k8s.io/customresourcedefinitions/, end=/registry/apiextensions.k8s.io/customresourcedefinitions/, revision=0, currentRev=0 count=0"
level=trace msg="COUNT /registry/events/, rev=0 => rev=0, count=0, err=<nil>, duration=592ns"
level=trace msg="LIST COUNT key=/registry/events/, end=/registry/events/, revision=0, currentRev=0 count=0"
level=trace msg="COUNT /registry/resourcequotas/, rev=0 => rev=0, count=0, err=<nil>, duration=290ns"
level=trace msg="LIST COUNT key=/registry/resourcequotas/, end=/registry/resourcequotas/, revision=0, currentRev=0 count=0"
level=trace msg="LIST /registry/resourcequotas/, start=/registry/resourcequotas/, limit=10001, rev=0 => rev=0, kvs=0, err=<nil>, duration=7.494µs"
level=trace msg="LIST key=/registry/resourcequotas/, end=/registry/resourcequotas/, revision=0, currentRev=0 count=0, limit=10000"
level=trace msg="COUNT /registry/secrets/, rev=0 => rev=0, count=0, err=<nil>, duration=310ns"
level=trace msg="LIST COUNT key=/registry/secrets/, end=/registry/secrets/, revision=0, currentRev=0 count=0"

jrovira-kumori commented 4 months ago

fwiw the bootstrap key is internal to k3s, I'm not sure this is the correct thing to create in kine.

The constant string "bootstrap" in the commits is meaningless. I believe any string would work here as the calls are done to the nats-go KeyValue api directly.

It should also have no effect as the value is immediately deleted.

brandond commented 4 months ago

level=error msg="Failed to create health check key: context deadline exceeded"

It sounds like the root cause here is that the health check key isn't getting created. Rather than creating and deleting another key, lets just fix the bug that's causing that key to not get created?

jrovira-kumori commented 4 months ago

It sounds like the root cause here is that the health check key isn't getting created. Rather than creating and deleting another key, lets just fix the bug that's causing that key to not get created?

The context deadline exceeded error was coming from github.com/nats-io/nats.go/jetstream@v1.31.0. The getMsg() wraps the context in a 5 second deadline when there is no other deadline defined.

I was unable to identify why the method times out after the stream is created in NATS. Maybe someone with more in-depth knowledge on its inner workings can clarify it. Anyhow, adding some retry logic to the Start() method seems to resolve the issues that were showing up.

I have removed the previous bootstrap key creation when initializing the NATS backend in favour of the retry logic.

jrovira-kumori commented 4 months ago

@brandond closed the PR by mistake...

bruth commented 4 months ago

Will take a look today.

bruth commented 4 months ago

The retry does work, but I am digging into why, once the KV bucket is created, the first write seemingly times out. It may be that the bucket is actually not yet ready to receive writes, but that should not be the case so I will dig a bit deeper on it.

brandond commented 4 months ago

@bruth any update on this? I don't think we should fix this on the kine side if the issue is in nats.

bruth commented 4 months ago

The latest version of the NATS Go client appears to fix the issue. At least I can't reproduce the bug with it. I will open a PR to update the NATS deps for Kine.

jrovira-kumori commented 4 months ago

The latest version of the NATS Go client appears to fix the issue. At least I can't reproduce the bug with it. I will open a PR to update the NATS deps for Kine.

I have tried the updated version from #281. From my testing, the issue still persists.

It can be reproduced easily with just nats-server (docker) and the kine binary. On a separate terminal run NATS:

docker run --rm --network host nats:2.10.11 -js -DVV

Then compile and run the updated kine version:

git clone https://github.com/nats-io/kine
cd kine
git switch update-nats-versions
go build .
./kine --debug --endpoint "nats://?noEmbed=true"

Kine will display the same error.

ERRO[2024-03-04T10:08:56.261135233+01:00] Failed to create health check key: context deadline exceeded

Full logs:

kine

``` DEBU[2024-03-04T10:08:51.255193539+01:00] using config &nats.Config{clientURL:"nats://", clientOptions:[]nats.Option(nil), revHistory:0xa, bucket:"kine", replicas:1, slowThreshold:500000000, noEmbed:false, dontListen:false, serverConfig:"", stdoutLogging:false, host:"", port:0, dataDir:""} INFO[2024-03-04T10:08:51.255250025+01:00] connecting to nats:// INFO[2024-03-04T10:08:51.255258441+01:00] using bucket: kine INFO[2024-03-04T10:08:51.25528472+01:00] metrics server is starting to listen at :8080 INFO[2024-03-04T10:08:51.255458986+01:00] starting metrics server path /metrics INFO[2024-03-04T10:08:51.259200075+01:00] bucket initialized: kine ERRO[2024-03-04T10:08:56.261135233+01:00] Failed to create health check key: context deadline exceeded INFO[2024-03-04T10:08:56.26128823+01:00] Kine available at http://127.0.0.1:2379 ^CERRO[2024-03-04T10:08:58.766961723+01:00] btree watcher error: context canceled ```

nats-server

``` [1] 2024/03/04 09:08:49.492112 [INF] Starting nats-server [1] 2024/03/04 09:08:49.492208 [INF] Version: 2.10.11 [1] 2024/03/04 09:08:49.492210 [INF] Git: [5fc6051] [1] 2024/03/04 09:08:49.492213 [DBG] Go build: go1.21.7 [1] 2024/03/04 09:08:49.492214 [INF] Name: NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H [1] 2024/03/04 09:08:49.492219 [INF] Node: fvNdmIeF [1] 2024/03/04 09:08:49.492221 [INF] ID: NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H [1] 2024/03/04 09:08:49.492241 [DBG] Created system account: "$SYS" [1] 2024/03/04 09:08:49.492375 [TRC] SYSTEM - <<- [SUB $SYS._INBOX.fvNdmIeF.* 2] [1] 2024/03/04 09:08:49.492403 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.ACCOUNT.*.CONNS 3] [1] 2024/03/04 09:08:49.492409 [TRC] SYSTEM - <<- [SUB $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H 4] [1] 2024/03/04 09:08:49.492418 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.NSUBS 5] [1] 2024/03/04 09:08:49.492422 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.STATSZ 6] [1] 2024/03/04 09:08:49.492425 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.SHUTDOWN 7] [1] 2024/03/04 09:08:49.492429 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.LAMEDUCK 8] [1] 2024/03/04 09:08:49.492433 [TRC] SYSTEM - <<- [SUB $SYS.ACCOUNT.*.CLAIMS.UPDATE 9] [1] 2024/03/04 09:08:49.492437 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE 10] [1] 2024/03/04 09:08:49.492442 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING 11] [1] 2024/03/04 09:08:49.492447 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.STATSZ 12] [1] 2024/03/04 09:08:49.492458 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.STATSZ 13] [1] 2024/03/04 09:08:49.492463 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.CONNZ 14] [1] 2024/03/04 09:08:49.492470 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.CONNZ 15] [1] 2024/03/04 09:08:49.492474 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.ROUTEZ 16] [1] 2024/03/04 09:08:49.492485 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.ROUTEZ 17] [1] 2024/03/04 09:08:49.492491 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.GATEWAYZ 18] [1] 2024/03/04 09:08:49.492498 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.GATEWAYZ 19] [1] 2024/03/04 09:08:49.492502 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.ACCOUNTZ 20] [1] 2024/03/04 09:08:49.492505 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.ACCOUNTZ 21] [1] 2024/03/04 09:08:49.492508 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.JSZ 22] [1] 2024/03/04 09:08:49.492512 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.JSZ 23] [1] 2024/03/04 09:08:49.492517 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.PROFILEZ 24] [1] 2024/03/04 09:08:49.492521 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.PROFILEZ 25] [1] 2024/03/04 09:08:49.492524 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.IDZ 26] [1] 2024/03/04 09:08:49.492528 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.IDZ 27] [1] 2024/03/04 09:08:49.492532 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.VARZ 28] [1] 2024/03/04 09:08:49.492536 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.VARZ 29] [1] 2024/03/04 09:08:49.492540 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.SUBSZ 30] [1] 2024/03/04 09:08:49.492546 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.SUBSZ 31] [1] 2024/03/04 09:08:49.492551 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.LEAFZ 32] [1] 2024/03/04 09:08:49.492554 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.LEAFZ 33] [1] 2024/03/04 09:08:49.492557 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.HEALTHZ 34] [1] 2024/03/04 09:08:49.492562 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.HEALTHZ 35] [1] 2024/03/04 09:08:49.492566 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.SUBSZ 36] [1] 2024/03/04 09:08:49.492571 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CONNZ 37] [1] 2024/03/04 09:08:49.492583 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.LEAFZ 38] [1] 2024/03/04 09:08:49.492587 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.JSZ 39] [1] 2024/03/04 09:08:49.492590 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.INFO 40] [1] 2024/03/04 09:08:49.492594 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.STATZ 41] [1] 2024/03/04 09:08:49.492600 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CONNS 42] [1] 2024/03/04 09:08:49.492596 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS 535] [1] 2024/03/04 09:08:49.492603 [TRC] SYSTEM - <<- [SUB $SYS.REQ.USER.*.INFO 43] [1] 2024/03/04 09:08:49.492614 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.PING.STATZ 44] [1] 2024/03/04 09:08:49.492619 [TRC] SYSTEM - <<- [SUB $SYS.ACCOUNT.*.LEAFNODE.CONNECT 45] [1] 2024/03/04 09:08:49.492623 [TRC] SYSTEM - <<- [SUB $SYS.LATENCY.M2.fvNdmIeF 46] [1] 2024/03/04 09:08:49.492607 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Q2\",\"timestamp\":\"2024-03-04T09:08:49.492278704Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":2,\"time\":\"2024-03-04T09:08:49.492405362Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":0,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:49.492626 [TRC] SYSTEM - <<- [SUB $SYS.DEBUG.SUBSCRIBERS 47] [1] 2024/03/04 09:08:49.492633 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.RELOAD 48] [1] 2024/03/04 09:08:49.492638 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.KICK 49] [1] 2024/03/04 09:08:49.492642 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.LDM 50] [1] 2024/03/04 09:08:49.492643 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS 535] [1] 2024/03/04 09:08:49.492649 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Q2\",\"timestamp\":\"2024-03-04T09:08:49.492278704Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":3,\"time\":\"2024-03-04T09:08:49.492634841Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":0,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:49.492668 [INF] Starting JetStream [1] 2024/03/04 09:08:49.492672 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS 535] [1] 2024/03/04 09:08:49.492675 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Rd\",\"timestamp\":\"2024-03-04T09:08:49.492392578Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":4,\"time\":\"2024-03-04T09:08:49.492664988Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":2,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:49.492690 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS 535] [1] 2024/03/04 09:08:49.492692 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Rd\",\"timestamp\":\"2024-03-04T09:08:49.492392578Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":5,\"time\":\"2024-03-04T09:08:49.492686588Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":2,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:49.492721 [TRC] SYSTEM - <<- [PUB $SYS.REQ.ACCOUNT.$G.CONNS $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H 261] [1] 2024/03/04 09:08:49.492724 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":6,\"time\":\"2024-03-04T09:08:49.492703761Z\"},\"acc\":\"$G\"}"] [1] 2024/03/04 09:08:49.492737 [TRC] SYSTEM - <<- [PUB $SYS.REQ.ACCOUNT.$SYS.CONNS $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H 263] [1] 2024/03/04 09:08:49.492741 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":7,\"time\":\"2024-03-04T09:08:49.492733206Z\"},\"acc\":\"$SYS\"}"] [1] 2024/03/04 09:08:49.492842 [DBG] JetStream creating dynamic configuration - 22.99 GB memory, 468.64 GB disk [1] 2024/03/04 09:08:49.493030 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __ [1] 2024/03/04 09:08:49.493036 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ | [1] 2024/03/04 09:08:49.493037 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| | [1] 2024/03/04 09:08:49.493039 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_| [1] 2024/03/04 09:08:49.493040 [INF] [1] 2024/03/04 09:08:49.493041 [INF] https://docs.nats.io/jetstream [1] 2024/03/04 09:08:49.493043 [INF] [1] 2024/03/04 09:08:49.493044 [INF] ---------------- JETSTREAM ---------------- [1] 2024/03/04 09:08:49.493047 [INF] Max Memory: 22.99 GB [1] 2024/03/04 09:08:49.493049 [INF] Max Storage: 468.64 GB [1] 2024/03/04 09:08:49.493051 [INF] Store Directory: "/tmp/nats/jetstream" [1] 2024/03/04 09:08:49.493053 [INF] ------------------------------------------- [1] 2024/03/04 09:08:49.493149 [TRC] SYSTEM - <<- [SUB $JS.API.> 51] [1] 2024/03/04 09:08:49.493225 [DBG] Exports: [1] 2024/03/04 09:08:49.493228 [DBG] $JS.API.> [1] 2024/03/04 09:08:49.493236 [TRC] SYSTEM - <<- [SUB $JS.API.ACCOUNT.PURGE.* 52] [1] 2024/03/04 09:08:49.493256 [DBG] Enabled JetStream for account "$G" [1] 2024/03/04 09:08:49.493263 [DBG] Max Memory: -1 B [1] 2024/03/04 09:08:49.493265 [DBG] Max Storage: -1 B [1] 2024/03/04 09:08:49.493487 [DBG] JetStream state for account "$G" recovered [1] 2024/03/04 09:08:49.493689 [INF] Listening for client connections on 0.0.0.0:4222 [1] 2024/03/04 09:08:49.493693 [DBG] Get non local IPs for "0.0.0.0" [1] 2024/03/04 09:08:49.493998 [DBG] ip=192.168.1.113 [1] 2024/03/04 09:08:49.494055 [DBG] ip=172.17.0.1 [1] 2024/03/04 09:08:49.494106 [DBG] ip=192.168.48.1 [1] 2024/03/04 09:08:49.494112 [DBG] ip=fc00:f853:ccd:e793::1 [1] 2024/03/04 09:08:49.494224 [INF] Server is ready [1] 2024/03/04 09:08:49.494415 [DBG] maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined [1] 2024/03/04 09:08:51.255553 [DBG] 127.0.0.1:49006 - cid:5 - Client connection created [1] 2024/03/04 09:08:51.255885 [TRC] 127.0.0.1:49006 - cid:5 - <<- [CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"kine using bucket: kine","lang":"go","version":"1.33.1","protocol":1,"echo":true,"headers":true,"no_responders":true}] [1] 2024/03/04 09:08:51.255987 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PING] [1] 2024/03/04 09:08:51.255991 [TRC] 127.0.0.1:49006 - cid:5 - ->> [PONG] [1] 2024/03/04 09:08:51.256112 [TRC] 127.0.0.1:49006 - cid:5 - <<- [SUB _INBOX.vDzLYVH9z9NsXOYvR0UmYH.* 1] [1] 2024/03/04 09:08:51.256126 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5B0kuxeZ 0] [1] 2024/03/04 09:08:51.256129 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""] [1] 2024/03/04 09:08:51.256530 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.DzKUmR 158] [1] 2024/03/04 09:08:51.256535 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"error\":{\"code\":404,\"err_code\":10059,\"description\":\"stream not found\"},\"total\":0,\"offset\":0,\"limit\":0}"] [1] 2024/03/04 09:08:51.256548 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5B0kuxeZ 1 158] [1] 2024/03/04 09:08:51.256553 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 692] [1] 2024/03/04 09:08:51.256555 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWL8\",\"timestamp\":\"2024-03-04T09:08:51.256489256Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"error\\\":{\\\"code\\\":404,\\\"err_code\\\":10059,\\\"description\\\":\\\"stream not found\\\"},\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0}\"}"] [1] 2024/03/04 09:08:51.256898 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.INFO _INBOX.vDzLYVH9z9NsXOYvR0UmYH.OZsWrv8O 0] [1] 2024/03/04 09:08:51.256904 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""] [1] 2024/03/04 09:08:51.257048 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.UoK1hf 363] [1] 2024/03/04 09:08:51.257052 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.account_info_response\",\"memory\":0,\"storage\":0,\"reserved_memory\":0,\"reserved_storage\":0,\"streams\":0,\"consumers\":0,\"limits\":{\"max_memory\":-1,\"max_storage\":-1,\"max_streams\":-1,\"max_consumers\":-1,\"max_ack_pending\":-1,\"memory_max_stream_bytes\":-1,\"storage_max_stream_bytes\":-1,\"max_bytes_required\":false},\"api\":{\"total\":1,\"errors\":1}}"] [1] 2024/03/04 09:08:51.257070 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.OZsWrv8O 1 363] [1] 2024/03/04 09:08:51.257075 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 902] [1] 2024/03/04 09:08:51.257077 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWQL\",\"timestamp\":\"2024-03-04T09:08:51.257028406Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"memory\\\":0,\\\"storage\\\":0,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"streams\\\":0,\\\"consumers\\\":0,\\\"limits\\\":{\\\"max_memory\\\":-1,\\\"max_storage\\\":-1,\\\"max_streams\\\":-1,\\\"max_consumers\\\":-1,\\\"max_ack_pending\\\":-1,\\\"memory_max_stream_bytes\\\":-1,\\\"storage_max_stream_bytes\\\":-1,\\\"max_bytes_required\\\":false},\\\"api\\\":{\\\"total\\\":1,\\\"errors\\\":1}}\"}"] [1] 2024/03/04 09:08:51.257280 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.CREATE.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.HzyzdaBy 422] [1] 2024/03/04 09:08:51.257289 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"discard\":\"old\",\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"deny_delete\":true,\"allow_rollup_hdrs\":true,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"consumer_limits\":{}}"] [1] 2024/03/04 09:08:51.257484 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS 536] [1] 2024/03/04 09:08:51.257492 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Up\",\"timestamp\":\"2024-03-04T09:08:51.257394773Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":8,\"time\":\"2024-03-04T09:08:51.257464002Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:51.257518 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS 536] [1] 2024/03/04 09:08:51.257521 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Up\",\"timestamp\":\"2024-03-04T09:08:51.257394773Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":9,\"time\":\"2024-03-04T09:08:51.257514206Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:51.258159 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.sSb33q 769] [1] 2024/03/04 09:08:51.258163 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_create_response\",\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"ts\":\"2024-03-04T09:08:51.258080497Z\",\"did_create\":true}"] [1] 2024/03/04 09:08:51.258193 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.HzyzdaBy 1 769] [1] 2024/03/04 09:08:51.258201 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1870] [1] 2024/03/04 09:08:51.258205 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWal\",\"timestamp\":\"2024-03-04T09:08:51.258131753Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.CREATE.KV_kine\",\"request\":\"{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"deny_delete\\\":true,\\\"allow_rollup_hdrs\\\":true,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"consumer_limits\\\":{}}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_create_response\\\",\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"ts\\\":\\\"2024-03-04T09:08:51.258080497Z\\\",\\\"did_create\\\":true}\"}"] [1] 2024/03/04 09:08:51.258367 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5rg88gvE 0] [1] 2024/03/04 09:08:51.258373 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""] [1] 2024/03/04 09:08:51.258457 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.h8FuzH 860] [1] 2024/03/04 09:08:51.258461 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"total\":0,\"offset\":0,\"limit\":0,\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"cluster\":{\"leader\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\"},\"ts\":\"2024-03-04T09:08:51.258417549Z\"}"] [1] 2024/03/04 09:08:51.258482 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5rg88gvE 1 860] [1] 2024/03/04 09:08:51.258487 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1479] [1] 2024/03/04 09:08:51.258488 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWfy\",\"timestamp\":\"2024-03-04T09:08:51.258435783Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0,\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"cluster\\\":{\\\"leader\\\":\\\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\\\"},\\\"ts\\\":\\\"2024-03-04T09:08:51.258417549Z\\\"}\"}"] [1] 2024/03/04 09:08:51.258613 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.UPDATE.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.UnFu5lTT 423] [1] 2024/03/04 09:08:51.258617 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"discard\":\"old\",\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"deny_delete\":true,\"allow_rollup_hdrs\":true,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"consumer_limits\":{}}"] [1] 2024/03/04 09:08:51.258710 [TRC] JETSTREAM - <-> [DELSUB 2] [1] 2024/03/04 09:08:51.258717 [TRC] JETSTREAM - <-> [DELSUB 3] [1] 2024/03/04 09:08:51.259032 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.zkwcsR 751] [1] 2024/03/04 09:08:51.259037 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_update_response\",\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"ts\":\"2024-03-04T09:08:51.25895733Z\"}"] [1] 2024/03/04 09:08:51.259063 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.UnFu5lTT 1 751] [1] 2024/03/04 09:08:51.259068 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1851] [1] 2024/03/04 09:08:51.259071 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWqO\",\"timestamp\":\"2024-03-04T09:08:51.259004909Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.UPDATE.KV_kine\",\"request\":\"{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"deny_delete\\\":true,\\\"allow_rollup_hdrs\\\":true,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"consumer_limits\\\":{}}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_update_response\\\",\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"ts\\\":\\\"2024-03-04T09:08:51.25895733Z\\\"}\"}"] [1] 2024/03/04 09:08:51.259368 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.DIRECT.GET.KV_kine.$KV.kine.L8nZchjdPLL.tzCUBDXD _INBOX.vDzLYVH9z9NsXOYvR0UmYH.3Xcl954a 0] [1] 2024/03/04 09:08:51.259376 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""] [1] 2024/03/04 09:08:51.259528 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.CONSUMER.CREATE.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1 _INBOX.vDzLYVH9z9NsXOYvR0UmYH.V0Q93c3O 221] [1] 2024/03/04 09:08:51.259553 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"stream_name\":\"KV_kine\",\"config\":{\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"deliver_policy\":\"all\",\"ack_policy\":\"none\",\"replay_policy\":\"instant\",\"inactive_threshold\":300000000000,\"num_replicas\":1,\"mem_storage\":true},\"action\":\"\"}"] [1] 2024/03/04 09:08:51.259860 [TRC] JETSTREAM - <<- [SUB $JSC.CI.$G.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1 53] [1] 2024/03/04 09:08:51.259882 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS 534] [1] 2024/03/04 09:08:51.259889 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55WQ\",\"timestamp\":\"2024-03-04T09:08:51.2598007Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":10,\"time\":\"2024-03-04T09:08:51.25987025Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:51.259923 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS 535] [1] 2024/03/04 09:08:51.259930 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55WQ\",\"timestamp\":\"2024-03-04T09:08:51.2598007Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":11,\"time\":\"2024-03-04T09:08:51.259913351Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"] [1] 2024/03/04 09:08:51.260087 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.1h0csz 582] [1] 2024/03/04 09:08:51.260092 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.consumer_create_response\",\"stream_name\":\"KV_kine\",\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"created\":\"2024-03-04T09:08:51.259797263Z\",\"config\":{\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"deliver_policy\":\"all\",\"ack_policy\":\"none\",\"max_deliver\":-1,\"replay_policy\":\"instant\",\"max_waiting\":512,\"inactive_threshold\":300000000000,\"num_replicas\":1,\"mem_storage\":true},\"delivered\":{\"consumer_seq\":0,\"stream_seq\":0},\"ack_floor\":{\"consumer_seq\":0,\"stream_seq\":0},\"num_ack_pending\":0,\"num_redelivered\":0,\"num_waiting\":0,\"num_pending\":0,\"ts\":\"2024-03-04T09:08:51.259876061Z\"}"] [1] 2024/03/04 09:08:51.260112 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.V0Q93c3O 1 582] [1] 2024/03/04 09:08:51.260119 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1459] [1] 2024/03/04 09:08:51.260122 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCX0o\",\"timestamp\":\"2024-03-04T09:08:51.260069994Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.CONSUMER.CREATE.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1\",\"request\":\"{\\\"stream_name\\\":\\\"KV_kine\\\",\\\"config\\\":{\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"deliver_policy\\\":\\\"all\\\",\\\"ack_policy\\\":\\\"none\\\",\\\"replay_policy\\\":\\\"instant\\\",\\\"inactive_threshold\\\":300000000000,\\\"num_replicas\\\":1,\\\"mem_storage\\\":true},\\\"action\\\":\\\"\\\"}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.consumer_create_response\\\",\\\"stream_name\\\":\\\"KV_kine\\\",\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"created\\\":\\\"2024-03-04T09:08:51.259797263Z\\\",\\\"config\\\":{\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"deliver_policy\\\":\\\"all\\\",\\\"ack_policy\\\":\\\"none\\\",\\\"max_deliver\\\":-1,\\\"replay_policy\\\":\\\"instant\\\",\\\"max_waiting\\\":512,\\\"inactive_threshold\\\":300000000000,\\\"num_replicas\\\":1,\\\"mem_storage\\\":true},\\\"delivered\\\":{\\\"consumer_seq\\\":0,\\\"stream_seq\\\":0},\\\"ack_floor\\\":{\\\"consumer_seq\\\":0,\\\"stream_seq\\\":0},\\\"num_ack_pending\\\":0,\\\"num_redelivered\\\":0,\\\"num_waiting\\\":0,\\\"num_pending\\\":0,\\\"ts\\\":\\\"2024-03-04T09:08:51.259876061Z\\\"}\"}"] [1] 2024/03/04 09:08:51.260416 [TRC] 127.0.0.1:49006 - cid:5 - <<- [SUB _INBOX.vDzLYVH9z9NsXOYvR0UmgZ 2] [1] 2024/03/04 09:08:51.260426 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.CONSUMER.MSG.NEXT.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1 _INBOX.vDzLYVH9z9NsXOYvR0UmgZ 64] [1] 2024/03/04 09:08:51.260430 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"expires\":30000000000,\"batch\":500,\"idle_heartbeat\":15000000000}"] [1] 2024/03/04 09:08:51.260461 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.VU63PUKW 0] [1] 2024/03/04 09:08:51.260466 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""] [1] 2024/03/04 09:08:51.260546 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.TdPzdJ 861] [1] 2024/03/04 09:08:51.260550 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"total\":0,\"offset\":0,\"limit\":0,\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":1},\"cluster\":{\"leader\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\"},\"ts\":\"2024-03-04T09:08:51.260516591Z\"}"] [1] 2024/03/04 09:08:51.260574 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.VU63PUKW 1 861] [1] 2024/03/04 09:08:51.260581 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API 1480] [1] 2024/03/04 09:08:51.260587 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCX61\",\"timestamp\":\"2024-03-04T09:08:51.260530607Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0,\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":1},\\\"cluster\\\":{\\\"leader\\\":\\\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\\\"},\\\"ts\\\":\\\"2024-03-04T09:08:51.260516591Z\\\"}\"}"] [1] 2024/03/04 09:08:53.498681 [DBG] 127.0.0.1:49006 - cid:5 - Client Ping Timer [1] 2024/03/04 09:08:53.498692 [TRC] 127.0.0.1:49006 - cid:5 - ->> [PING] [1] 2024/03/04 09:08:53.498860 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PONG] [1] 2024/03/04 09:08:58.768966 [DBG] 127.0.0.1:49006 - cid:5 - Client connection closed: Client Closed [1] 2024/03/04 09:08:58.769016 [TRC] 127.0.0.1:49006 - cid:5 - <-> [DELSUB 1] [1] 2024/03/04 09:08:58.769026 [TRC] 127.0.0.1:49006 - cid:5 - <-> [DELSUB 2] ^C[1] 2024/03/04 09:08:59.651387 [DBG] Trapped "interrupt" signal [1] 2024/03/04 09:08:59.701672 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.SHUTDOWN 240] [1] 2024/03/04 09:08:59.701689 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":12,\"time\":\"2024-03-04T09:08:59.701632213Z\"}"] [1] 2024/03/04 09:08:59.701728 [INF] Initiating Shutdown... [1] 2024/03/04 09:08:59.701745 [INF] Initiating JetStream Shutdown... [1] 2024/03/04 09:08:59.701791 [DBG] JETSTREAM - JetStream connection closed: Client Closed [1] 2024/03/04 09:08:59.701807 [DBG] JETSTREAM - JetStream connection closed: Client Closed [1] 2024/03/04 09:08:59.701819 [DBG] JETSTREAM - JetStream connection closed: Client Closed [1] 2024/03/04 09:08:59.701830 [DBG] JETSTREAM - JetStream connection closed: Client Closed [1] 2024/03/04 09:08:59.701891 [INF] JetStream Shutdown [1] 2024/03/04 09:08:59.701899 [DBG] JETSTREAM - JetStream connection closed: Client Closed [1] 2024/03/04 09:08:59.701957 [DBG] Client accept loop exiting.. [1] 2024/03/04 09:08:59.701946 [DBG] SYSTEM - System connection closed: Client Closed [1] 2024/03/04 09:08:59.701982 [INF] Server Exiting.. ```

kannanvr commented 3 months ago

Any update on this ? We are also facing the similar issue

bruth commented 3 months ago

Found the root cause and updated my previous PR: https://github.com/k3s-io/kine/pull/281. @jrovira-kumori feel free to validate on your end.

jrovira-kumori commented 3 months ago

I am happy to confirm that the issue has been resolved! I have tested it again with kube-apiserver (just in case) and it runs like a charm.

Thanks for the help!

k3s-io / kine

NATS: Fix illegal resource version from storage: 0 (#274) #275