search

peterprib / node-red-contrib-kafka-manager

Implement
GNU General Public License v3.0
22 stars 12 forks source link

OffsetOutOfRange #33

Closed tompeel closed 3 years ago

tompeel commented 3 years ago

Using the kafka consumer node I'm getting the error OffsetOutOfRrang (PAUSED).

I can see the offset is 331104153 using another kafka tool out side of node red.

If I inject msg.action:fetch into the kafka offset node it crashes the web ui of node red and i have to refresh my browser after a few seconds.

In the kafka admin node when i request the topic i get [{"0":{"nodeId":0,"host":"####","port":###},"4":{"nodeId":4,"host":"####","port":####},"12":{"nodeId":12,"host":"####","port":###}},{"metadata":{"Mdss-Binary":{"0":{"topic":"Mdss-Binary","partition":0,"leader":4,"replicas":[],"isr":[]}},"Mdss-Json":{"0":{"topic":"Mdss-Json","partition":0,"leader":12,"replicas":[],"isr":[]}}},"clusterMetadata":{"controllerId":0}}]

So i assume i have a good connection and the auth and credentials are correct.

How can i debug this problem further?

peterprib commented 3 years ago

If you send log I can workout what may be causing crash of web ui. This sounds odd as not sure what would induce.

Actually problem see to be related to https://github.com/SOHU-Co/kafka-node/issues/170

tompeel commented 3 years ago

Which particulart logs are you after, im using node red in homeassistant at the moment, so i probably need to drill down in there.

If I use the Kafka Offset node I get the error "kafka offset signal not deployed"

In relation to the SOHU-Co/kadka-node#170 I'm not sure how this applies, obviously the offset number I'm working with is huge some 30 million or so and counting up all the time...

The consumer note does pause when you get the out of range error.

I'm not all that experienced on kafka, what should my offset be in the consumer node if i just want to pull the most recent 10000 datapoints?

peterprib commented 3 years ago

The node.js logs at point of failure. 30M isn't that big although near 32bit max which may being used and causing an issue at some layer. Logs should highlight if this is the case.

peterprib commented 3 years ago

Closing as after a bit more research determine the first problem is most likely to be at Kafka layer. The second problem being Web UI crashing is not likely to be caused by code in this module but values pasted that may not be handled by node-red. Browser log should help to identify root cause of issue. Can't do much more on either without extracts of relevant logs.