search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.31k stars 1.24k forks source link

Running Pinot with an invalid zookeeper service name seems to error during clean-up (in addition to connection error) #5802

Open lgo opened 4 years ago

lgo commented 4 years ago

Running with an invalid zookeeper service name seems to have an error during clean-up in addition to the main error, eg: bin/pinot-admin.sh StartController -clusterName PinotCluster -controllerHost null -controllerPort 9000 -dataDir /tmp/PinotController -zkAddress pinotzookeeper.service:2181

It appears that not only did it error on java.net.UnknownHostException: pinotzookeeper.service: Name or service not known, but it also error'd on afterwards (or during recovery). 

java.lang.NullPointerException: null
    at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112)

Looks like it just may not be skipping cleanup when no connection was established?

Here's the full stack trace.

Initiating client connection, connectString=pinotzookeeper.service:2181 sessionTimeout=30000 watcher=org.apache.helix.manager.zk.client.ZkConnectionManager@432038ec
Unable to connect to server: pinotzookeeper:2181
java.net.UnknownHostException: pinotzookeeper.service: Name or service not known
    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_252]
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_252]
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_252]
    at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_252]
    at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_252]
    at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_252]
    at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:62) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:449) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:383) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.zookeeper.ZkConnection.connect(ZkConnection.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1565) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:179) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.ZKHelixAdmin.<init>(ZKHelixAdmin.java:107) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
Failed to start a [ CONTROLLER ] Service
java.lang.NullPointerException: null
    at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1591) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:179) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.helix.manager.zk.ZKHelixAdmin.<init>(ZKHelixAdmin.java:107) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixClusterIfNeeded(HelixSetupUtils.java:69) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.helix.core.util.HelixSetupUtils.setupHelixController(HelixSetupUtils.java:63) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.ControllerStarter.setUpHelixController(ControllerStarter.java:310) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:289) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:113) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:90) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:200) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:195) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:166) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
    at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-a8fbdaeffaf99c1df4258f60ecbbab1f553d7696]
xiangfu0 commented 3 years ago

This will error out during start up as controller cannot connect to zookeeper for further processing. And you can safely ignore this helix cleanup as there is no connection to zookeeper hence no cleanup is required.