linkedin / Burrow

Kafka Consumer Lag Checking
Apache License 2.0
3.73k stars 795 forks source link

Burrow returning empty topic/consumer lists #514

Closed thefunkjunky closed 5 years ago

thefunkjunky commented 5 years ago

After giving up on using our cluster SASL-Plaintext authentication/encryption (see https://github.com/linkedin/Burrow/issues/380#issuecomment-480385599), I've tested burrow with simple plaintext (no encryption/authentication) to confirm that it can connect at all. It does seem to connect without error,

{"level":"info","ts":1554492061.736778,"msg":"Started Burrow"}
{"level":"info","ts":1554492061.7368414,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.7371376,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1554492061.7371557,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1554492061.7372136,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1554492061.7372262,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1554492061.7372491,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1554492061.746594,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1554492061.7466223,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1554492061.7466445,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"myclustername"}
{"level":"info","ts":1554492061.7618148,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1554492061.7618647,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"myconsumers"}
{"level":"info","ts":1554492061.7975335,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.8112516,"msg":"Connected to 52.27.94.190:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.815502,"msg":"Authenticated: id=101861002933334153, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.8155353,"msg":"Re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.820379,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1554492061.820402,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1554492061.8204606,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1554492061.820469,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1554492061.8204806,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1554492061.8206592,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8443"}
{"level":"info","ts":1554492061.820674,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1554492061.820687,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1554492061.8206909,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"myclustername"}

but returns empty lists of topics and consumers when queried:

$ curl https://my.burrow:8443/v3/kafka/myclustername/topic
{"error":false,"message":"topic list returned","topics":[],"request":{"url":"/v3/kafka/myclustername/topic","host":"my.burrow"}}

$ curl https://my.burrow:8443/v3/kafka/myclustername/consumer
{"error":false,"message":"consumer list returned","consumers":[],"request":{"url":"/v3/kafka/myclustername/consumer","host":"my.burrow"}}

My config:

[general]
pidfile="/var/lock/burrow/burrow.pid"
stdout-logfile="burrow.out"

[zookeeper]
servers=[
  "my.zookeeper:2181"
]
timeout=6
root-path="/burrow"

[client-profile.myclient]
kafka-version="0.9.0"
client-id="burrow-myclient"
tls="mytlsprofile"

[tls.mytlsprofile]
certfile="whatever.crt"
keyfile="whatever.key"
cafile="/etc/pki/tls/certs/ca-bundle.crt"
noverify=true

[httpserver.tlslistener]
address=":8443"
timeout=300
tls="mytlsprofile"

[cluster.myclustername]
class-name="kafka"
servers=[
  "my.kafka:9092"
]
client-profile="myclient"
topic-refresh=120
offset-refresh=30

[consumer.myconsumers]
class-name="kafka"
cluster="myclustername"
servers=[
  "my.kafka:9092"
]
client-profile="myclient"
offsets-topic="__consumer_offsets"
start-latest=true
group-whitelist=".*"

Kafka ACLs:

Current ACLs for resource `Topic:LITERAL:__consumer_offsets`:
  User:* has Allow permission for operations: All from hosts: *
  User:ANONYMOUS has Allow permission for operations: All from hosts: *

Current ACLs for resource `Group:LITERAL:*`:
  User:* has Allow permission for operations: All from hosts: *
  User:ANONYMOUS has Allow permission for operations: All from hosts: *

Current ACLs for resource `Topic:LITERAL:*`:
  User:ANONYMOUS has Allow permission for operations: Read from hosts: *

Using confluent kafka 2.0.0 (which I believe is kafka v. 0.9.0 or 0.9.0.0). I have tried this with various combinations of the above settings (such as noverify set to true or false), kafka 0.9.0 vs 0.9.0.0, direction connections to zookeeper/kafka vs. AWS ELBs, etc. Nothing seems to work

thefunkjunky commented 5 years ago

closing. Turns out that there was no plaintext listen set up on this cluster, despite the fact that burrow 'successfully' started up and connected to it. I may open a separate ticket for that, not sure if that's something that warrants one