search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.48k stars 1.28k forks source link

bug: broker crash at startup with `zip file closed` #6669

Open Eywek opened 3 years ago

Eywek commented 3 years ago

I'm trying to up a pinot cluster for my tests with the following docker-compose configuration:

version: '3'
services:
  zookeeper:
    image: zookeeper:3.5.6

  pinot:
    image: apachepinot/pinot:0.6.0
    command: StartServiceManager -clusterName e2e -port 9001 -zkAddress zookeeper:2181 -bootstrapConfigPaths /var/pinot/config/controller.conf /var/pinot/config/broker.conf /var/pinot/config/server.conf
    volumes:
      - "./packages/e2e/apache-pinot:/var/pinot/config"
    depends_on:
      - zookeeper
    # ports:
    #   - 9000:9000

broker.conf:

pinot.service.role=BROKER
pinot.broker.client.queryPort=8099
pinot.broker.routing.table.builder.class=random

controller.conf:

pinot.service.role=CONTROLLER
controller.port=9000
controller.vip.host=pinot-controller
controller.vip.port=9000
controller.data.dir=/var/pinot/controller/data
controller.zk.str=zookeeper:2181

server.conf:

pinot.service.role=SERVER
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment

But sometimes, the broker fail to start with the following error in logs:

2021-03-11T14:22:02.1567273Z pinot_1        | Failed to start a Pinot [BROKER] at 24.895 since launch
2021-03-11T14:22:02.1568445Z pinot_1        | java.lang.IllegalStateException: zip file closed
2021-03-11T14:22:02.1569607Z pinot_1        |   at java.util.zip.ZipFile.ensureOpen(ZipFile.java:686) ~[?:1.8.0_275]
2021-03-11T14:22:02.1570744Z pinot_1        |   at java.util.zip.ZipFile.access$200(ZipFile.java:60) ~[?:1.8.0_275]
2021-03-11T14:22:02.1571931Z pinot_1        |   at java.util.zip.ZipFile$ZipEntryIterator.hasNext(ZipFile.java:508) ~[?:1.8.0_275]
2021-03-11T14:22:02.1573506Z pinot_1        |   at java.util.zip.ZipFile$ZipEntryIterator.hasMoreElements(ZipFile.java:503) ~[?:1.8.0_275]
2021-03-11T14:22:02.1575015Z pinot_1        |   at java.util.jar.JarFile$JarEntryIterator.hasNext(JarFile.java:253) ~[?:1.8.0_275]
2021-03-11T14:22:02.1576425Z pinot_1        |   at java.util.jar.JarFile$JarEntryIterator.hasMoreElements(JarFile.java:262) ~[?:1.8.0_275]
2021-03-11T14:22:02.1578434Z pinot_1        |   at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:30) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1580893Z pinot_1        |   at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:26) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1584116Z pinot_1        |   at shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1587855Z pinot_1        |   at shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1590879Z pinot_1        |   at org.reflections.Reflections.scan(Reflections.java:243) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1593449Z pinot_1        |   at org.reflections.Reflections.scan(Reflections.java:202) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1596113Z pinot_1        |   at org.reflections.Reflections.<init>(Reflections.java:123) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1598881Z pinot_1        |   at io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1601766Z pinot_1        |   at io.swagger.jaxrs.config.BeanConfig.scanAndRead(BeanConfig.java:240) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1604651Z pinot_1        |   at io.swagger.jaxrs.config.BeanConfig.setScan(BeanConfig.java:221) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1608446Z pinot_1        |   at org.apache.pinot.broker.broker.BrokerAdminApiApplication.setupSwagger(BrokerAdminApiApplication.java:77) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1612534Z pinot_1        |   at org.apache.pinot.broker.broker.BrokerAdminApiApplication.start(BrokerAdminApiApplication.java:65) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1616505Z pinot_1        |   at org.apache.pinot.broker.broker.helix.HelixBrokerStarter.start(HelixBrokerStarter.java:248) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1620225Z pinot_1        |   at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:135) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1624058Z pinot_1        |   at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1628130Z pinot_1        |   at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1632631Z pinot_1        |   at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1637612Z pinot_1        |   at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]
2021-03-11T14:22:02.1641976Z pinot_1        |   at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]

Leading to an error when I try to create a table: Failed to find instances with tag: DefaultTenant_BROKER for table...

xiangfu0 commented 3 years ago

hmm, can you try to start pinot-controller/broker/server separately in your docker compose? Meanwhile, I will take a look at the issue.