search

yahoo / CMAK

CMAK is a tool for managing Apache Kafka clusters
Apache License 2.0
11.84k stars 2.5k forks source link

Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again. #748

Open lifa1225 opened 4 years ago

lifa1225 commented 4 years ago

hi,how to fix it？ image image

bjosve commented 4 years ago

I had the same error. When I replaced the config files with the old 2.0.0.2 config files everything worked again. Maybe something is broken with the new "KMScheduleLeaderElectionFeature"?!

j-mie commented 4 years ago

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

Getting this in the logs:

Mar 11 13:55:44 ngkafka-1 cmak[1194]: 2020-03-11 13:55:44,760 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:54 ngkafka-1 cmak[1194]: 2020-03-11 13:55:54,759 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,157 - [INFO] o.a.z.ClientCnxn - Unable to read additional data from server sessionid 0x30000a360a60001, likely server has closed socket, closing socket connection and attempting reconnect
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,169 - [ERROR] k.m.KafkaManager - Failed on input : KMAddCluster(ClusterConfig(vantage,CuratorConfig(ngkafka-1:2181,ngkafka-2:2181,ngkafka-3:2181,ngkafka-4:2181,ngkafka-5:2181,10,100,1000),true,2.4.0,false,None,None,false,false,false,false,false,false,Some(ClusterTuning(Some(30),Some(2),Some(100),Some(2),Some(100),Some(2),Some(100),Some(30),Some(5),Some(2),Some(1000),Some(2),Some(1000),Some(2),Some(1000),Some(30000),Some(1000000),Some(7))),PLAINTEXT,None,None))
Mar 11 13:55:57 ngkafka-1 cmak[1194]: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:746)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,178 - [ERROR] k.m.ApiError$ - error : KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:746)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,066 - [INFO] o.a.z.ClientCnxn - Opening socket connection to server ngkafka-2/10.20.4.90:2181. Will not attempt to authenticate using SASL (unknown error)
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,067 - [INFO] o.a.z.ClientCnxn - Socket connection established, initiating session, client: /10.20.4.89:51838, server: ngkafka-2/10.20.4.90:2181
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,070 - [INFO] o.a.z.ClientCnxn - Session establishment complete on server ngkafka-2/10.20.4.90:2181, sessionid = 0x30000a360a60001, negotiated timeout = 40000
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,072 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,073 - [INFO] k.m.a.KafkaManagerActor - Pruning clusters...
Mar 11 13:56:04 ngkafka-1 cmak[1194]: 2020-03-11 13:56:04,761 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:56:14 ngkafka-1 cmak[1194]: 2020-03-11 13:56:14,759 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...

In our Zookeeper logs I'm seeing:

2020-03-11 13:59:54,825 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@756] - Received packet at server of unknown type 19
2020-03-11 13:59:54,826 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.20.4.89:48398 which had sessionid 0x40000a3500a0001
bjosve commented 4 years ago

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

We are running CMAK 3.0.0.4 (docker image hlebalbau/kafka-manager) and Kafka 2.0.1 (docker image wurstmeister/kafka). I replaced both "application.conf" and "routes" with the old config and no more errors.

j-mie commented 4 years ago

Updating our Zookeeper version from 3.4.14 to 3.5.7 fixed this for us

priyanka460 commented 4 years ago

@jamie-34254 how to update zookeeper version or how to replace config file with old one?

priyanka460 commented 4 years ago

I had the same error. When I replaced the config files with the old 2.0.0.2 config files everything worked again. Maybe something is broken with the new "KMScheduleLeaderElectionFeature"?!

how to do that?

priyanka460 commented 4 years ago

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

We are running CMAK 3.0.0.4 (docker image hlebalbau/kafka-manager) and Kafka 2.0.1 (docker image wurstmeister/kafka). I replaced both "application.conf" and "routes" with the old config and no more errors.

from where you found old config file

gl1ridae commented 4 years ago

I have the same issue, and seems like it's related with unsupported zookeeper version. Since latest CMAK use Apache Curator 2.12.0 and it supported zookeeper 3.5.x version by default, and 3.4.x only with modifications in build.

It will be nice to add Zookeeper requirements in README.

godwhoa commented 4 years ago

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"
rohitranjanef commented 4 years ago

I have the same problem. I have installed zookeeper and kafka with docker images. with zookeeper v3.5.8 I get KeeperErrorCode = Unimplemented for /kafka-manager/mutex when I try to create a cluster. Any help would be appreciated.

zhigang commented 4 years ago

create /kafka-manager/mutex/locks and /kafka-manager/mutex/leases to fixed this, like this: https://github.com/yahoo/CMAK/issues/731#issuecomment-643880544

TangZhongham commented 4 years ago

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"

it works! thanks a lot

wyisterl commented 4 years ago

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"

It works, but maybe need some attention to "org.apache.zookeeper" % "zookeeper" % "3.4.14",,3.4.14 refering your zk version maybe? I modify my zk version 3.4.10 instead, and recompile it. Thanks a lot!

Zhangwusheng commented 3 years ago

create /kafka-manager/mutex/locks and /kafka-manager/mutex/leases to fixed this, like this: #731 (comment)

works for me . thank you

jaypalsodha commented 2 years ago

org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /debug

I am facing the same issue. Above resolutions shows that we are forcing zookeeper version 3.4.14 to be used. But my requirement is to use zookeeper version 3.5.7 with curator-framework version 2.12.0 or latest.

Any help would be appreciated.

ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Ensure path threw exception org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /debug at org.apache.zookeeper.KeeperException.create(KeeperException.java:106) [CCB-globalCronExecutor-3-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x17fb5659c8f0056, likely server has closed socket, closing socket connection and attempting reconnect at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291) at org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) at org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83) at org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594) at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158) at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) at com.santaba.server.util.cloud.Register2CloudZKTask._CreateDiagPathInZookeeper(Register2CloudZKTask.java:160) at com.santaba.server.util.cloud.Register2CloudZKTask._run(Register2CloudZKTask.java:177) at com.santaba.server.cron.Cron.run(Cron.java:72) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [CCB-globalCronExecutor-3-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED [2022-03-23 06:44:02.013 UTC] [MSG] [WARN] [CCB-globalCronExecutor-3:santaba:cron:] [Register2CloudZKTask._CreateDiagPathInZookeeper:163] something wrong when create the path of diagnosis, CONTEXT=connectString=zookeeper:2181, rootPath=debug, errMsg=KeeperErrorCode = ConnectionLoss for /debug

wyisterl commented 2 years ago

这是来自QQ邮箱的自动回复邮件。   您好，我已经收到你的邮件

papillon88 commented 2 years ago

"org.apache.zookeeper" % "zookeeper" % "3.4.14"

I removed that line and it still worked

wyisterl commented 2 years ago

这是来自QQ邮箱的自动回复邮件。   您好，我已经收到你的邮件