friedrichwilken commented 1 year ago

Description

Update to the latest version of the NATS Charts including the NATS Server, Config Reloader and Prometheus Exporter images.

Config Reloader and Prometheus Exporter was updated in this PR.

https://github.com/nats-io/nats-server/releases/tag/v2.10.7

Acceptance

[x] update to the latest version of NATS
[x] use latest version of official helm charts
[x] run load-tests for the new version
[x] ~~update documentation (referenced versions)~~
[x] run upgrade tests (deploy old version of nats run loadtests and upgrade to the new version (under load))
- [x] use production like cluster (3 k8s nodes etc.)
- [x] run loadtest with more than 500 eps

kyma-bot commented 1 year ago

This issue or PR has been automatically marked as stale due to the lack of recent activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

After 60d of inactivity, lifecycle/stale is applied
After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

kyma-bot commented 1 year ago

This issue or PR has been automatically closed due to the lack of activity. Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

After 60d of inactivity, lifecycle/stale is applied
After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle stale

If you think that I work incorrectly, kindly raise an issue with the problem.

/close

kyma-bot commented 1 year ago

@kyma-bot: Closing this issue.

In response to [this](https://github.com/kyma-project/nats-manager/issues/104#issuecomment-1805697400): >This issue or PR has been automatically closed due to the lack of activity. >Thank you for your contributions. > >This bot triages issues and PRs according to the following rules: >- After 60d of inactivity, `lifecycle/stale` is applied >- After 7d of inactivity since `lifecycle/stale` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle stale` > >If you think that I work incorrectly, kindly [raise an issue](https://github.com/kyma-project/test-infra/issues/new/choose) with the problem. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

mfaizanse commented 10 months ago

https://github.com/nats-io/k8s/blob/main/helm/charts/nats/UPGRADING.md

mfaizanse commented 10 months ago

NATS 2.10.7 tests

Summary

Main differences in Helm Chart

A new K8s Cluster IP Service is introduced for the clients to connect to NATS.
- The old headless service is renamed to xyz-nats-headless, and it is only supposed to be used by NATS Servers for cluster route connections (reference).
- In NATS server config, cluster_advertise is removed and no_advertise: true is set.
The NATS server configurations is changed to JSON format in configmap.yaml.
There are some minor changes as done in this PR.

Results from manual testing of NATS 2.10.7
Load tests (with CE.v64 and CE.v128) on new Eventing cluster with NATS 2.10.7, ran fine. Kept the cluster running for 3 days.
Upgrade tests from NATS 2.9.18 to 2.10.7 with load tests running had some issues, sometimes the NATS cluster got into broken state with the error message failed to be ready for connections after 1ms: server, route in NATS server logs.
- Out of 5 upgrade tests, 4 were fine.
Sometimes deleting a subscription with high load of events published may cause the NATS servers to become unready with message: JetStream is not current with the meta leader
- Subscription status showed: failed to delete JetStream subscription: failed to delete consumer: context deadline exceeded

Running load tests on new cluster with NATS 2.10.7

With the following Subscriptions till CE.v64: OKAY
With the following Subscriptions till CE.v128: OKAY

Upgrade from NATS 2.9.18 to 2.10.7

Steps followed:

Create a new Eventing cluster with NATS 2.9.18.
Deploy load tests.
Upgrade the cluster to use NATS 2.10.7.
Stop the load tests to check if all events are delivered or not.

Upgrade test 1:

eventing-nats-0                             3/3     Running   0                 2h
eventing-nats-1                             2/3     Running   0                 2m
eventing-nats-2                             2/3     Running   0                 5m

# logs eventing-nats-2
[33] 2024/01/03 14:33:21.354816 [INF] ---------------- JETSTREAM ----------------
[33] 2024/01/03 14:33:21.354823 [INF]   Max Memory:      1.00 GB
[33] 2024/01/03 14:33:21.354826 [INF]   Max Storage:     1.00 GB
[33] 2024/01/03 14:33:21.354828 [INF]   Store Directory: "/data/jetstream"
[33] 2024/01/03 14:33:21.354830 [INF] -------------------------------------------
[33] 2024/01/03 14:33:21.356478 [INF]   Starting restore for stream '$G > sap'
[33] 2024/01/03 14:33:21.358283 [INF]   Restored 90,371 messages for stream '$G > sap' in 2ms
[33] 2024/01/03 14:33:21.358339 [INF]   Recovering 14 consumers for stream - '$G > sap'
[33] 2024/01/03 14:33:40.300862 [WRN] Healthcheck failed: "failed to be ready for connections after 1ms: server, route"
[33] 2024/01/03 14:33:50.302951 [WRN] Healthcheck failed: "failed to be ready for connections after 1ms: server, route"
# Please edit the object below. Lines beginning with a '#' will be ignored,
[33] 2024/01/03 14:34:00.301214 [WRN] Healthcheck failed: "failed to be ready for connections after 1ms: server, route"
[33] 2024/01/03 14:34:10.301385 [WRN] Healthcheck failed: "failed to be ready for connections after 1ms: server, route"

Upgrade test 2:

With load tests CE.v64 , there are some missing events.

There were some NACKs as well in the logs.

Deleting the load tests worked fine.

loadtest-subscriber-0-cdc46c975-zvh2l loadtest-subscriber 2024-01-05T10:25:35.CE.v8: TotalMissing: 6 Min: 0 Max: 2,895 Min Missing: 2,212 Missing: [[2,212] [2,226] [2,228 -> 2,229] [2,233] [2,240]]
loadtest-subscriber-1-cf654fcd-52pbz loadtest-subscriber 2024-01-05T10:25:35.LEGACY.v8: TotalMissing: 8 Min: 8 Max: 2,895 Min Missing: 2,208 Missing: [[2,208] [2,210] [2,212 -> 2,215] [2,229] [2,231]]

## stream info
Cluster Information:

                 Name: eventing-nats
               Leader: eventing-nats-2
              Replica: eventing-nats-0, current, seen 0.19s ago
              Replica: eventing-nats-1, current, seen 0.19s ago

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 129,638
              LastSeq: 129,637 @ 2024-01-05T10:31:36 UTC
     Active Consumers: 8

# ╰─ k logs -n kyma-system eventing-manager-79678cb7c8-6l5lc| grep -i failed
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"73477a81-bc26-45f6-b6fa-f408835e5caa","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"d6db7c2d-0613-4b4e-96e4-6c116d64b852","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"f7b0a698-8641-42be-b7c5-1377a0a2fdf1","source":"cloudevent","type":"kyma.cloudevent.CE.v16","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"fb50db63-668d-478d-b0ad-22e65d4d0b14","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"ad5fd537-62ce-4c66-869c-e66ed6198649","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:35Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"04f94bef-6715-4b02-a1a5-b346cd5d2631","source":"cloudevent","type":"kyma.cloudevent.CE.v16","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:36Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"d09cfe72-36a2-4b16-a9c9-fa0f92c360ef","source":"cloudevent","type":"kyma.cloudevent.CE.v16","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T10:25:36Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"5104ac6e-ed33-402e-947e-66048c0fb998","source":"cloudevent","type":"kyma.cloudevent.CE.v16","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.145.210:80: connect: connection refused"}}

Upgrade test 3:

with load tests CE.v128 , there are some missing events.

There were some NACKs as well in the logs.

Deleting the load tests worked fine.

loadtest-subscriber-1-cf654fcd-pc4p7 loadtest-subscriber 2024-01-05T12:17:22.LEGACY.v8: TotalMissing: 10 Min: 16 Max: 4,431 Min Missing: 3,249 Missing: [[3,249] [3,258 -> 3,259] [3,262] [3,264] [3,266] [3,269 -> 3,270] [3,275] [3,280]]
loadtest-subscriber-0-cdc46c975-mw9pm loadtest-subscriber 2024-01-05T12:17:22.CE.v8: TotalMissing: 9 Min: 0 Max: 4,431 Min Missing: 3,248 Missing: [[3,248 -> 3,249] [3,251 -> 3,252] [3,254 -> 3,257] [3,261]]

## stream info
Cluster Information:

                 Name: eventing-nats
               Leader: eventing-nats-2
              Replica: eventing-nats-0, current, seen 0.23s ago
              Replica: eventing-nats-1, current, seen 0.23s ago

State:

             Messages: 1
                Bytes: 503 B
             FirstSeq: 276,531 @ 2024-01-05T12:23:33 UTC
              LastSeq: 410,057 @ 2024-01-05T12:26:35 UTC
     Deleted Messages: 133,526
     Active Consumers: 10
   Number of Subjects: 1

# ╰─ k logs -n kyma-system eventing-manager-79678cb7c8-prwf9 | grep -i failed
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"13f8768d-2eea-4b1d-948c-1a7f2eb15211","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"27ea285f-0499-4855-a28d-0149fb026034","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"f9ad5128-a652-4384-9966-73f02564007c","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"3c6af4b0-ff3b-42a8-9b12-13382bac7634","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"a9925e78-f687-4d82-85d3-1fa4be06358f","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T12:17:22Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"4d357b31-d927-4d26-976c-b5bf0de1f763","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.111.147.206:80: connect: connection refused"}}

Upgrade test 4:

with load tests CE.v128

Deleting the load tests worked fine.

loadtest-subscriber-0-cdc46c975-q8rb7 loadtest-subscriber 2024-01-05T13:29:07.CE.v8: TotalMissing: 0 Min: 0 Max: 4,175 Min Missing: 0 Missing: []
loadtest-subscriber-1-cf654fcd-chp85 loadtest-subscriber 2024-01-05T13:29:07.LEGACY.v8: TotalMissing: 0 Min: 16 Max: 4,175 Min Missing: 0 Missing: []

## stream info
Cluster Information:

                 Name: eventing-nats
               Leader: eventing-nats-1
              Replica: eventing-nats-0, current, seen 0.39s ago
              Replica: eventing-nats-2, current, seen 0.39s ago

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 388,043
              LastSeq: 388,042 @ 2024-01-05T13:37:48 UTC
     Active Consumers: 10

# ╰─ k logs -n kyma-system eventing-manager-79678cb7c8-wtt2j | grep -i failed
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"73d9c63d-e21b-474a-b754-dea401a0c6df","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"1e2ccff1-f963-41ce-8819-60991ff27c4c","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"ed1a95b7-7e83-4292-bdb7-965f3ab958fd","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"5c8b2f2a-178f-49e9-92f5-12dc57da11ed","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"5c46653a-414a-4c2c-822e-75c3c0a078c0","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:29:07Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"85c55af1-586c-4e68-9da7-8416dde6cbd5","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.107.234.65:80: connect: connection refused"}}

Upgrade test 5:

with load tests CE.v128

loadtest-subscriber-0-cdc46c975-lwj2k loadtest-subscriber 2024-01-05T13:43:45.CE.v8: TotalMissing: 0 Min: 0 Max: 3,991 Min Missing: 0 Missing: []
loadtest-subscriber-1-cf654fcd-dtcl4 loadtest-subscriber 2024-01-05T13:43:45.LEGACY.v8: TotalMissing: 0 Min: 16 Max: 3,991 Min Missing: 0 Missing: []

## stream info
Cluster Information:

                 Name: eventing-nats
               Leader: eventing-nats-2
              Replica: eventing-nats-0, current, seen 0.91s ago
              Replica: eventing-nats-1, current, seen 0.91s ago

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 371,235
              LastSeq: 371,234 @ 2024-01-05T13:52:03 UTC
     Active Consumers: 10

# ╰─ k logs -n kyma-system eventing-manager-79678cb7c8-jm8gr | grep -i failed
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"aa58ef95-da8d-474b-aa94-dc1c59c5a737","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"9fe7ef6a-ee41-406c-a7f3-1dac6aa8c70f","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"be80467f-78a2-4062-b379-13092d6ae33b","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"d267fffa-4b79-43c9-958d-9bb75c2edbd9","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"8a908398-71bc-4851-86ad-eb8c3ed43107","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"c43e1492-f2d8-4722-8ab1-e577cb851ece","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}
{"level":"ERROR","timestamp":"2024-01-05T13:43:45Z","logger":"jetstream-handler","caller":"jetstream/jetstream.go:549","message":"Failed to dispatch the CloudEvent","context":{"id":"c3b17437-2472-438d-b1dc-1b66ea6862f5","source":"cloudevent","type":"kyma.cloudevent.CE.v8","sink":"http://loadtest-subscriber-0.eventing-test.svc.cluster.local","error":"Post \"http://loadtest-subscriber-0.eventing-test.svc.cluster.local\": dial tcp 100.105.109.250:80: connect: connection refused"}}

Cluster not ready when deleting load tests subscriptions

Deploy loadtests.
Wait 1-2 minutes
Delete loadtests.

The subscriptions will get stuck in deletion phase with the following message in status:

conditions:
  - lastTransitionTime: "2024-01-05T09:29:55Z"
    message: 'failed to delete JetStream subscription: failed to delete consumer:
      context deadline exceeded, consumer: 2c377e9d82b8f81a248a3c6360ced0cc'
    reason: NATS Subscription not active
    status: "False"
    type: Subscription active

eventing-nats-0                             3/3     Running   0                 24h
eventing-nats-1                             2/3     Running   0                 24h
eventing-nats-2                             2/3     Running   0                 24

# logs eventing-nats-2
[32] 2024/01/05 09:31:25.028419 [WRN] RAFT [LQ7xh9M7 - _meta_] Falling behind in health check, commit 17 != applied 16
[32] 2024/01/05 09:31:25.028446 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"

# logs eventing-nats-1
[32] 2024/01/05 09:31:55.066741 [WRN] RAFT [oaVGX5Kl - _meta_] Falling behind in health check, commit 17 != applied 16
[32] 2024/01/05 09:31:55.066767 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"

mfaizanse commented 10 months ago

Summary after testing again on Azure with same machine type as in Prod:

Tested the NATS upgrade from NATS-manager 1.0.2 to PR-273
Pending messages before the NATS version upgrade were delivered after the upgrade.
- Steps: Create subscription → scaledown sink → publish events → upgrade NATS → scaled up sink —> events were delivered

Upgrade:

Test 1 with 512 eps: OKAY
Test 2 with 512 eps: OKAY
Test 3 with 512 eps: OKAY

Test 4 with 1024 eps: OKAY

The first NATS pod with new NATS image took 16 minutes to get ready. It was not ready 2/3 for 16 minutes and then the pod was restarted and became ready 3/3.

eventing-nats-2                             3/3     Running     1 (67m ago)   82m

# eventing-nats-2 logs when it was not ready
[33] 2024/01/09 12:48:02.389888 [WRN] Consumer create failed, could not locate stream '$G > sap > d43ab2220b90f7f85d635170e31ce8d4'
[33] 2024/01/09 12:48:02.389895 [WRN] Healthcheck failed: "JetStream stream '$G > sap' is not current"

# nats stream info sap
Cluster Information:

                 Name: eventing-nats
               Leader: eventing-nats-0
              Replica: eventing-nats-1, current, seen 0.00s ago
              Replica: eventing-nats-2, outdated, seen 12m58s ago, 79,502 operations behind

State:

             Messages: 355
                Bytes: 177 KiB
             FirstSeq: 1,393,829 @ 2024-01-09T12:47:51 UTC
              LastSeq: 1,394,323 @ 2024-01-09T12:47:51 UTC
     Deleted Messages: 140
     Active Consumers: 2
   Number of Subjects: 2

Test 5 with 2048 eps: OKAY
- First NATS pod took 3 minutes to become ready 3/3 without any pod restart.

kyma-project / nats-manager

Update NATS Server Version 2.10.7 #104

Description

Acceptance

NATS 2.10.7 tests

Summary

Main differences in Helm Chart

Results from manual testing of NATS 2.10.7

Running load tests on new cluster with NATS 2.10.7

With the following Subscriptions till `CE.v128`: `OKAY`

Upgrade from NATS 2.9.18 to 2.10.7

Steps followed:

Upgrade test 1:

Upgrade test 2:

Upgrade test 3:

Upgrade test 4:

Upgrade test 5:

Cluster not ready when deleting load tests subscriptions

Summary after testing again on Azure with same machine type as in Prod:

kyma-project / nats-manager

Update NATS Server Version 2.10.7 #104

Description

Acceptance

NATS 2.10.7 tests

Summary

Main differences in Helm Chart

Results from manual testing of NATS 2.10.7

Running load tests on new cluster with NATS 2.10.7

With the following Subscriptions till CE.v128: OKAY

Upgrade from NATS 2.9.18 to 2.10.7

Steps followed:

Upgrade test 1:

Upgrade test 2:

Upgrade test 3:

Upgrade test 4:

Upgrade test 5:

Cluster not ready when deleting load tests subscriptions

Summary after testing again on Azure with same machine type as in Prod:

With the following Subscriptions till `CE.v128`: `OKAY`