search

redpanda-data / helm-charts

Redpanda Helm Chart
http://redpanda.com
Apache License 2.0
79 stars 97 forks source link

Missing Kafka topics in Console #928

Open Aransh opened 11 months ago

Aransh commented 11 months ago

What happened?

While console summary shows a large number of messages (over 50000), the console view shows "no messages". 04_21_49

What did you expect to happen?

I expected the messages in the topic to appear.

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

```console apiVersion: cluster.redpanda.com/v1alpha1 kind: Redpanda metadata: name: redpanda namespace: redpanda spec: chartRef: {} clusterSpec: auth: sasl: enabled: true mechanism: SCRAM-SHA-512 secretRef: kafka-credentials users: [] connectors: enabled: true monitoring: enabled: true console: ingress: annotations: cert-manager.io/cluster-issuer: letsencrypt-production className: nginx enabled: true monitoring: enabled: true resources: cpu: cores: 4 memory: container: max: 10Gi min: 10Gi enable_memory_locking: true statefulset: initContainers: configurator: resources: limits: cpu: 1 memory: 1Gi setDataDirOwnership: enabled: true tuning: resources: limits: cpu: 1 memory: 1Gi replicas: 3 ```

Anything else we need to know?

This doesn't happen with all topics. In this scenario I onlyy spotted this on this topics, while other topics on the same cluster were showing fine. Also got reports of this from other team members, so not local cache.

Which are the affected charts?

Redpanda, console

Chart Version(s)

```console operator - 0.4.8 redpanda - 5.6.58 ```

Cloud provider

Linode

JIRA Link: K8S-85

alejandroEsc commented 11 months ago

@Aransh could we please get a few logs from console and from chrome so we can spot what may be causing this. It could be a console specific issue which we will then want to file an issue in that repo.

Aransh commented 11 months ago

@alejandroEsc sure, attaching logs, let me know if this is what you meant. Keep in mind these logs are for the same topic from the screenshot above, which now has 1,479,084 messages

Console logs:

{"level":"info","ts":"2023-12-17T06:01:48.053Z","msg":"started Redpanda Console","version":"v2.3.8","built_at":"1701900386"} {"level":"info","ts":"2023-12-17T06:01:48.054Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"} {"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"successfully connected to kafka cluster","advertised_broker_count":3,"topic_count":124,"controller_id":2,"kafka_version":"unknown custom version at least v0.11.0"} {"level":"info","ts":"2023-12-17T06:01:48.072Z","msg":"creating schema registry client and testing connectivity"} {"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"successfully tested schema registry connectivity"} {"level":"info","ts":"2023-12-17T06:01:48.080Z","msg":"testing admin client connectivity","urls":["https://redpanda.redpanda.svc.cluster.local.:9644"]} {"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"successfully tested the Redpanda admin connectivity","broker_count":3,"cluster_version":"Redpanda v23.2.18"} {"level":"info","ts":"2023-12-17T06:01:48.085Z","msg":"creating Kafka connect HTTP clients and testing connectivity to all clusters"} {"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"tested Kafka connect cluster connectivity","successful_clusters":1,"failed_clusters":0} {"level":"info","ts":"2023-12-17T06:01:48.203Z","msg":"successfully create Kafka connect service"} {"level":"info","ts":"2023-12-17T06:01:48.552Z","msg":"Server listening on address","address":"[::]:8080","port":8080} {"level":"info","ts":"2023-12-19T15:27:53.044Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"} {"level":"info","ts":"2023-12-19T15:27:53.053Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[140{51104 e-1 ce0} 293{50249 e-1 ce0} 292{49480 e-1 ce0} 159{49877 e-1 ce0} 0{50357 e-1 ce0} 82{49623 e-1 ce0} 107{50965 e-1 ce0} 9{50211 e-1 ce0} 133{50918 e-1 ce0} 118{49965 e-1 ce0} 250{50739 e-1 ce0} 235{50713 e-1 ce0} 100{49808 e-1 ce0} 246{49802 e-1 ce0} 79{51005 e-1 ce0} 200{49205 e-1 ce0} 150{50020 e-1 ce0} 213{50298 e-1 ce0} 50{50158 e-1 ce0} 183{50868 e-1 ce0} 265{50622 e-1 ce0} 153{50753 e-1 ce0} 199{50186 e-1 ce0} 174{50767 e-1 ce0} 254{50182 e-1 ce0} 123{50133 e-1 ce0} 102{50285 e-1 ce0} 204{50470 e-1 ce0} 16{51032 e-1 ce0} 28{49938 e-1 ce0} 229{49721 e-1 ce0} 248{50189 e-1 ce0} 68{50869 e-1 ce0} 195{50339 e-1 ce0} 70{49004 e-1 ce0} 141{50227 e-1 ce0} 60{50616 e-1 ce0} 167{50556 e-1 ce0} 42{50713 e-1 ce0} 242{50165 e-1 ce0} 117{50363 e-1 ce0} 148{50217 e-1 ce0} 162{50532 e-1 ce0} 270{51117 e-1 ce0} 171{49189 e-1 ce0} 147{49414 e-1 ce0} 1{50720 e-1 ce0} 95{50340 e-1 ce0} 273{49482 e-1 ce0} 297{50799 e-1 ce0}]"} {"level":"info","ts":"2023-12-19T15:29:01.411Z","msg":"immediate metadata update triggered","source":"kafka_client","why":"querying metadata for consumer initialization"} {"level":"info","ts":"2023-12-19T15:29:01.421Z","msg":"assigning partitions","source":"kafka_client","why":"new assignments from direct consumer","how":"assigning everything new, keeping current assignment","input":"csi_lke_perf_test.yondu-waf-triggers-dlr[235{50713 e-1 ce0} 91{49567 e-1 ce0} 261{49194 e-1 ce0} 76{50671 e-1 ce0} 238{50588 e-1 ce0} 131{49856 e-1 ce0} 109{49256 e-1 ce0} 46{50303 e-1 ce0} 61{49607 e-1 ce0} 263{49895 e-1 ce0} 40{50722 e-1 ce0} 262{49779 e-1 ce0} 287{51348 e-1 ce0} 35{48972 e-1 ce0} 44{49782 e-1 ce0} 144{49280 e-1 ce0} 200{49215 e-1 ce0} 250{50743 e-1 ce0} 4{48956 e-1 ce0} 251{49158 e-1 ce0} 104{50021 e-1 ce0} 94{49577 e-1 ce0} 228{51204 e-1 ce0} 213{50298 e-1 ce0} 272{49607 e-1 ce0} 63{50595 e-1 ce0} 93{49488 e-1 ce0} 102{50295 e-1 ce0} 137{50336 e-1 ce0} 60{50616 e-1 ce0} 71{50542 e-1 ce0} 237{50431 e-1 ce0} 165{49282 e-1 ce0} 8{50656 e-1 ce0} 170{51337 e-1 ce0} 217{50290 e-1 ce0} 160{50615 e-1 ce0} 119{49886 e-1 ce0} 283{49622 e-1 ce0} 214{51024 e-1 ce0} 75{51208 e-1 ce0} 166{50870 e-1 ce0} 248{50189 e-1 ce0} 37{49643 e-1 ce0} 142{50168 e-1 ce0} 154{50145 e-1 ce0} 183{50878 e-1 ce0} 70{49004 e-1 ce0} 121{50442 e-1 ce0} 259{49912 e-1 ce0}]"}

Chrome console:

GET https://console.<domain-redacted>/api/users/me 405 (Method Not Allowed)

executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 _callee3$ @ backendApi.ts:258 tryCatch @ regeneratorRuntime.js:44 (anonymous) @ regeneratorRuntime.js:125 (anonymous) @ regeneratorRuntime.js:69 asyncGeneratorStep @ asyncToGenerator.js:3 _next @ asyncToGenerator.js:22 (anonymous) @ asyncToGenerator.js:27 (anonymous) @ asyncToGenerator.js:19 refreshUserData @ backendApi.ts:257 executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 UserProfile @ UserButton.tsx:29 Eh @ react-dom.profiling.min.js:162 Mj @ react-dom.profiling.min.js:215 Jl @ react-dom.profiling.min.js:322 Il @ react-dom.profiling.min.js:306 Hl @ react-dom.profiling.min.js:305 vl @ react-dom.profiling.min.js:305 tl @ react-dom.profiling.min.js:292 J @ scheduler.production.min.js:13 R @ scheduler.production.min.js:14

Backend reported new/unknown endpoints for endpointCompatibility: "GET /api/users"

(anonymous) @ supportedFeatures.ts:66 executeAction @ action.ts:69 When-effect @ action.ts:50 (anonymous) @ when.ts:73 reactionRunner @ autorun.ts:88 trackDerivedFunction @ derivation.ts:183 track @ reaction.ts:137 (anonymous) @ autorun.ts:59 runReaction @ reaction.ts:102 runReactionsHelper @ reaction.ts:260 reactionScheduler @ reaction.ts:230 (anonymous) @ reaction.ts:270 El @ react-dom.profiling.min.js:298 reactionScheduler @ reaction.ts:270 runReactions @ reaction.ts:237 endBatch @ observable.ts:112 reportChanged @ atom.ts:76 setNewValue @ observablevalue.ts:141 setObservablePropValue @ observableobject.ts:166 set @ observableobject.ts:198 set @ dynamicobject.ts:42 _callee12$ @ backendApi.ts:699 tryCatch @ regeneratorRuntime.js:44 (anonymous) @ regeneratorRuntime.js:125 (anonymous) @ regeneratorRuntime.js:69 asyncGeneratorStep @ asyncToGenerator.js:3 _next @ asyncToGenerator.js:22 Promise.then (async) asyncGeneratorStep @ asyncToGenerator.js:12 _next @ asyncToGenerator.js:22 (anonymous) @ asyncToGenerator.js:27 (anonymous) @ asyncToGenerator.js:19 refreshSupportedEndpoints @ backendApi.ts:695 executeAction @ action.ts:69 ObservableObject.key @ action.ts:50 (anonymous) @ config.ts:163 memoized @ memoize-one.esm.js:40 App @ App.tsx:51 (anonymous) @ observer.ts:104 (anonymous) @ useObserver.ts:126 trackDerivedFunction @ derivation.ts:183 track @ reaction.ts:137 useObserver @ useObserver.ts:124 observerComponent @ observer.ts:104 li @ react-dom.profiling.min.js:182 Hj @ react-dom.profiling.min.js:214 Gj @ react-dom.profiling.min.js:212 Ej @ react-dom.profiling.min.js:211 Jl @ react-dom.profiling.min.js:327 Il @ react-dom.profiling.min.js:306 Hl @ react-dom.profiling.min.js:305 vl @ react-dom.profiling.min.js:305 tl @ react-dom.profiling.min.js:292 J @ scheduler.production.min.js:13 R @ scheduler.production.min.js:14