ZK error: invalid ACL specified

[theduderog]

I'm getting this error trying to run burrow.

Started Burrow at June 12, 2015 at 11:14am (PDT)
1434132884352394945 [Info] Starting Zookeeper client
1434132884352448792 [Info] Starting Offsets Storage module
1434132884352593141 [Info] Starting HTTP server
1434132884352628654 [Info] Starting Zookeeper client for cluster sit
1434132884352649043 [Info] Starting Kafka client for cluster sit
1434132884837543547 [Info] Starting consumers for 50 partitions of __consumer_offsets in cluster sit
1434132906887302495 [Critical] Cannot get ZK notifier lock: zk: invalid ACL specified

This is my config

[general]
logdir=log
logconfig=logging.cfg
pidfile=burrow.pid
client-id=burrow-lagchecker
group-blacklist=^(console-consumer-|python-kafka-consumer-).*$

[zookeeper]
hostname=host1
hostname=host2
hostname=host3
port=2181
timeout=6
lock-path=/burrow/notifier

[kafka "sit"]
broker=host1
broker=host2
broker=host3
broker=host4
broker-port=9092
zookeeper=host1
zookeeper=host2
zookeeper=host3
zookeeper-port=2181
zookeeper-path=/
offsets-topic=__consumer_offsets

[tickers]
broker-offsets=60

[lagcheck]
intervals=10
expire-group=604800

[httpserver]
server=on
port=7000

I'm using ZK 3.4.6 with no ACLs.

[theduderog]

This might be due to me not having any new consumers that are saving their offsets in Kafka. What's the easiest way to test this out?

[toddpalino]

Even though you don't have new consumers, it does appear that you have a __consumer_offsets topic from your log. This ZK error is being generated when Burrow tries to create and use the /burrow/notifier path in Zookeeper in order to lock against multiple instances of Burrow running at the same time (so that only one of them runs notifiers). You don't have any notifiers configured, so strictly speaking we could just skip this step.

However, you still should not see an error like this, even with no notifiers configured. I need to dig into the ZK client's lock implementation to see where this error could be coming from.

[toddpalino]

Can you please try changing line 180 in main.go to this and running?

appContext.NotifierLock = zk.NewLock(zkconn, appContext.Config.Zookeeper.LockPath, zk.WorldACL(zk.PermAll))

[theduderog]

Thank you, Todd. I can't get on the network until Monday but I'll try this as soon as I can.

Sent from my iPhone

On Jun 13, 2015, at 9:47 AM, toddpalino notifications@github.com wrote:

Can you please try changing line 180 in main.go to this and running?

appContext.NotifierLock = zk.NewLock(zkconn, appContext.Config.Zookeeper.LockPath, zk.WorldACL(zk.PermAll)) — Reply to this email directly or view it on GitHub.

[philip-sterne]

I ran into this issue, and changing line 180 as above fixed it. Thanks!

[toddpalino]

Thanks, @philip-sterne! I'm going to leave this open for @theduderog to confirm as well. It looks like it's a subtle difference in the way ACL definitions are handled between the version of ZK that you guys have and what we're running internally. It's possible ours is not exactly 3.4.6 because of the way we are building it.

I'm going to commit this change shortly.

[splee]

I just ran into this issue myself, and I think I must have installed about 10 minutes before your commit :)

Pulling from master fixed this for me also.

[theduderog]

Sorry for the delay. Will confirm tomorrow am.

On Mon, Jun 15, 2015 at 10:13 AM, toddpalino notifications@github.com wrote:

Thanks, @philip-sterne https://github.com/philip-sterne! I'm going to leave this open for @theduderog https://github.com/theduderog to confirm as well. It looks like it's a subtle difference in the way ACL definitions are handled between the version of ZK that you guys have and what we're running internally. It's possible ours is not exactly 3.4.6 because of the way we are building it.

I'm going to commit this change shortly.

— Reply to this email directly or view it on GitHub https://github.com/linkedin/Burrow/issues/3#issuecomment-112142619.

[theduderog]

It worked. Thank you!

Now I see this:

1434470206834270789 [Info] Acquired Zookeeper notifier lock