search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.39k stars 3.68k forks source link

Failure to ingest local file when running through the tutorial with the docker cluster #8827

Closed scottbelden closed 4 years ago

scottbelden commented 4 years ago

Affected Version

0.16.0-incubating

Description

I am using the docker-compose file located here to start a druid cluster. I modified the compose file to specify the 0.16.0-incubating tag for the druid images.

I am trying to follow the tutorial to ingest a local file but after submitting the job it never succeeds and instead always fails.

Below are the docker-compose logs for the cluster from the time I submitted the task to the time that it failed:


overlord         | 2019-11-05T22:54:22,055 INFO [qtp2107873140-89] org.apache.druid.indexing.overlord.TaskLockbox - Adding task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] to activeTasks
overlord         | 2019-11-05T22:54:22,055 INFO [TaskQueue-Manager] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - [forceTimeChunkLock] is set to true in task context. Use timeChunk lock
overlord         | 2019-11-05T22:54:22,056 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z
overlord         | 2019-11-05T22:54:22,056 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z
overlord         | 2019-11-05T22:54:22,058 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[172.19.0.6:8091] to add task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z]
overlord         | 2019-11-05T22:54:22,061 DEBUG [qtp2107873140-89] org.apache.druid.jetty.RequestLog - 172.19.0.7 POST //172.19.0.4:8081/druid/indexer/v1/task HTTP/1.1
router           | 2019-11-05T22:54:22,063 DEBUG [AsyncManagementForwardingServlet-2b1cd7bc-177] org.apache.druid.jetty.RequestLog - 172.19.0.1 POST //localhost:4008/druid/indexer/v1/task HTTP/1.1
overlord         | 2019-11-05T22:54:22,076 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z switched from pending to running (on [172.19.0.6:8091])
overlord         | 2019-11-05T22:54:22,077 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [RUNNING].
middlemanager    | 2019-11-05T22:54:22,092 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] started.
middlemanager    | 2019-11-05T22:54:22,095 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Running command: java -cp /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/jackson-databind-2.6.7.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/commons-lang-2.6.jar:lib/google-oauth-client-1.22.0.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/log4j-1.2-api-2.5.jar:lib/rhino-1.7.11.jar:lib/maven-model-builder-3.1.1.jar:lib/joda-time-2.10.2.jar:lib/druid-console-0.16.0-incubating.jar:lib/zookeeper-3.4.14.jar:lib/maven-repository-metadata-3.1.1.jar:lib/avatica-server-1.12.0.jar:lib/jsr311-api-1.1.1.jar:lib/jackson-jaxrs-json-provider-2.6.7.jar:lib/jetty-servlet-9.4.10.v20180503.jar:lib/druid-sql-0.16.0-incubating.jar:lib/druid-indexing-hadoop-0.16.0-incubating.jar:lib/checker-qual-2.5.7.jar:lib/reactive-streams-1.0.2.jar:lib/derbynet-10.14.2.0.jar:lib/protobuf-java-3.1.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/json-path-2.3.0.jar:lib/netty-3.10.6.Final.jar:lib/esri-geometry-api-2.0.0.jar:lib/jackson-datatype-guava-2.6.7.jar:lib/aws-java-sdk-kms-1.11.199.jar:lib/asm-analysis-7.1.jar:lib/jboss-logging-3.1.3.GA.jar:lib/hibernate-validator-5.1.3.Final.jar:lib/jackson-module-guice-2.6.7.jar:lib/druid-services-0.16.0-incubating.jar:lib/guice-4.1.0.jar:lib/jackson-annotations-2.6.7.jar:lib/netty-handler-4.1.30.Final.jar:lib/netty-codec-http-4.1.30.Final.jar:lib/commons-math3-3.6.1.jar:lib/jackson-jq-0.0.10.jar:lib/guice-multibindings-4.1.0.jar:lib/netty-buffer-4.1.30.Final.jar:lib/spymemcached-2.12.3.jar:lib/javax.inject-1.jar:lib/jetty-security-9.4.10.v20180503.jar:lib/aws-java-sdk-ec2-1.11.199.jar:lib/config-magic-0.9.jar:lib/maven-settings-builder-3.1.1.jar:lib/wagon-provider-api-2.4.jar:lib/curator-client-4.1.0.jar:lib/jackson-dataformat-cbor-2.6.7.jar:lib/curator-recipes-4.1.0.jar:lib/guice-servlet-4.1.0.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/audience-annotations-0.5.0.jar:lib/avatica-metrics-1.12.0.jar:lib/netty-handler-proxy-4.1.30.Final.jar:lib/joni-2.1.27.jar:lib/xz-1.8.jar:lib/metrics-core-4.0.0.jar:lib/ion-java-1.0.2.jar:lib/javax.el-api-3.0.0.jar:lib/jcodings-1.0.43.jar:lib/jetty-client-9.4.10.v20180503.jar:lib/commons-dbcp2-2.0.1.jar:lib/curator-framework-4.1.0.jar:lib/commons-text-1.3.jar:lib/netty-reactive-streams-2.0.0.jar:lib/stax-ex-1.8.jar:lib/druid-server-0.16.0-incubating.jar:lib/netty-codec-4.1.30.Final.jar:lib/druid-gcp-common-0.16.0-incubating.jar:lib/commons-compress-1.18.jar:lib/janino-2.7.6.jar:lib/jetty-util-9.4.10.v20180503.jar:lib/slf4j-api-1.7.25.jar:lib/netty-transport-native-epoll-4.1.30.Final-linux-x86_64.jar:lib/jsr305-2.0.1.jar:lib/icu4j-54.1.1.jar:lib/accessors-smart-1.2.jar:lib/error_prone_annotations-2.3.2.jar:lib/commons-logging-1.1.1.jar:lib/jetty-continuation-9.4.10.v20180503.jar:lib/jersey-server-1.19.3.jar:lib/shims-0.8.6.jar:lib/commons-net-3.6.jar:lib/httpcore-4.4.11.jar:lib/commons-io-2.6.jar:lib/javax.activation-api-1.2.0.jar:lib/netty-common-4.1.30.Final.jar:lib/okhttp-1.0.2.jar:lib/fastutil-8.2.3.jar:lib/jersey-guice-1.19.3.jar:lib/jersey-servlet-1.19.3.jar:lib/jvm-attach-api-1.5.jar:lib/jackson-datatype-joda-2.6.7.jar:lib/jline-0.9.94.jar:lib/FastInfoset-1.2.15.jar:lib/aws-java-sdk-s3-1.11.199.jar:lib/sigar-1.6.5.132.jar:lib/netty-transport-4.1.30.Final.jar:lib/jackson-dataformat-smile-2.6.7.jar:lib/commons-codec-1.12.jar:lib/netty-resolver-dns-4.1.30.Final.jar:lib/opencsv-4.6.jar:lib/log4j-slf4j-impl-2.5.jar:lib/jackson-core-asl-1.9.13.jar:lib/derby-10.14.2.0.jar:lib/druid-hll-0.16.0-incubating.jar:lib/guava-16.0.1.jar:lib/jdbi-2.63.1.jar:lib/antlr4-runtime-4.5.1.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/RoaringBitmap-0.8.6.jar:lib/validation-api-1.1.0.Final.jar:lib/disruptor-3.3.6.jar:lib/json-smart-2.3.jar:lib/netty-transport-native-unix-common-4.1.30.Final.jar:lib/aopalliance-1.0.jar:lib/jackson-module-jaxb-annotations-2.6.7.jar:lib/google-api-client-1.22.0.jar:lib/calcite-core-1.17.0.jar:lib/lz4-java-1.6.0.jar:lib/jmespath-java-1.11.199.jar:lib/aether-util-0.9.0.M2.jar:lib/druid-aws-common-0.16.0-incubating.jar:lib/derbyclient-10.14.2.0.jar:lib/jackson-jaxrs-base-2.6.7.jar:lib/commons-lang3-3.8.1.jar:lib/jetty-http-9.4.10.v20180503.jar:lib/calcite-linq4j-1.17.0.jar:lib/commons-collections4-4.2.jar:lib/jna-4.5.1.jar:lib/compress-lzf-1.0.4.jar:lib/jaxb-api-2.3.1.jar:lib/log4j-core-2.5.jar:lib/tesla-aether-0.0.5.jar:lib/aggdesigner-algorithm-6.0.jar:lib/jetty-io-9.4.10.v20180503.jar:lib/aws-java-sdk-core-1.11.199.jar:lib/druid-processing-0.16.0-incubating.jar:lib/javax.activation-1.2.0.jar:lib/commons-collections-3.2.2.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/jackson-jaxrs-smile-provider-2.6.7.jar:lib/maven-aether-provider-3.1.1.jar:lib/asm-7.1.jar:lib/txw2-2.3.1.jar:lib/maven-settings-3.1.1.jar:lib/asm-commons-7.1.jar:lib/jetty-servlets-9.4.10.v20180503.jar:lib/asm-tree-7.1.jar:lib/aether-spi-0.9.0.M2.jar:lib/plexus-utils-3.0.15.jar:lib/caffeine-2.5.5.jar:lib/netty-codec-dns-4.1.30.Final.jar:lib/netty-codec-socks-4.1.30.Final.jar:lib/log4j-jul-2.5.jar:lib/httpclient-4.5.3.jar:lib/jaxb-runtime-2.3.1.jar:lib/async-http-client-2.5.3.jar:lib/airline-0.7.jar:lib/classmate-1.0.0.jar:lib/zstd-jni-1.3.3-1.jar:lib/extendedset-0.16.0-incubating.jar:lib/commons-pool2-2.2.jar:lib/druid-core-0.16.0-incubating.jar:lib/log4j-api-2.5.jar:lib/javax.el-3.0.0.jar:lib/aether-impl-0.9.0.M2.jar:lib/jetty-server-9.4.10.v20180503.jar:lib/javax.servlet-api-3.1.0.jar:lib/curator-x-discovery-4.1.0.jar:lib/jackson-core-2.6.7.jar:lib/jersey-core-1.19.3.jar:lib/avatica-core-1.12.0.jar:lib/commons-beanutils-1.9.3.jar:lib/netty-resolver-4.1.30.Final.jar:lib/google-http-client-jackson2-1.22.0.jar:lib/maven-artifact-3.6.0.jar:lib/commons-compiler-2.7.6.jar:lib/maven-model-3.1.1.jar:lib/plexus-interpolation-1.19.jar:lib/druid-indexing-service-0.16.0-incubating.jar:lib/google-http-client-1.22.0.jar:lib/aether-api-0.9.0.M2.jar:lib/jetty-proxy-9.4.10.v20180503.jar: -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -server -Xmx1g -Xms1g -XX:MaxDirectMemorySize=3g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Ddruid.indexer.task.baseTaskDir=var/druid/task -Ddruid.host=172.19.0.6 -Ddruid.metadata.storage.host= -Ddruid.metadata.storage.connector.password=FoolishPassword -Ddruid.metadata.storage.connector.host=localhost -Ddruid.indexer.fork.property.druid.processing.numThreads=1 -Ddruid.emitter.logging.logLevel=debug -Ddruid.emitter=noop -Ddruid.azure.key=YOURKEY -Ddruid.indexer.fork.property.druid.processing.buffer.sizeBytes=268435456 -Duser.timezone=UTC -Dfile.encoding.pkg=sun.io -Ddruid.azure.maxTries=3 -Ddruid.storage.storageDirectory=var/druid/segments -Ddruid.selectors.coordinator.serviceName=druid/coordinator -Ddruid.selectors.indexing.serviceName=druid/overlord -Ddruid.indexing.doubleStorage=double -Ddruid.lookup.enableLookupSyncOnStartup=false -Ddruid.server.http.numThreads=60 -Ddruid.worker.capacity=4 -Ddruid.metadata.storage.connector.port=1527 -Ddruid.azure.container=druid -Ddruid.azure.account=YOURACCOUNT -Ddruid.service=druid/middleManager -Ddruid.metadata.storage.connector.user=druid -Ddruid.metadata.storage.type=postgresql -Ddruid.metadata.storage.connector.connectURI=jdbc:postgresql://postgres:5432/druid -Ddruid.coordinator.balancer.strategy=cachingCost -Ddruid.plaintextPort=8091 -Djava.io.tmpdir=var/tmp -Ddruid.azure.protocol=https -Ddruid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "druid-azure-extensions", "postgresql-metadata-storage"] -Ddruid.sql.enable=true -Ddruid.startup.logging.logProperties=true -Ddruid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"] -Ddruid.zk.service.host=zookeeper -Ddruid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"] -Ddruid.indexer.logs.directory=var/druid/indexing-logs -Ddruid.zk.paths.base=/druid -Dfile.encoding=UTF-8 -Ddruid.storage.type=azure -Ddruid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp -Ddruid.indexer.fork.property.druid.processing.numMergeBuffers=2 -Ddruid.indexer.logs.type=file -Ddruid.processing.numThreads=1 -Ddruid.processing.buffer.sizeBytes=268435456 -Ddruid.processing.numMergeBuffers=2 -Ddruid.metrics.emitter.dimension.dataSource=wikiticker-2015-09-12-sampled -Ddruid.metrics.emitter.dimension.taskId=index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z -Ddruid.metrics.emitter.dimension.taskType=index_parallel -Ddruid.host=172.19.0.6 -Ddruid.plaintextPort=8100 -Ddruid.tlsPort=-1 -Ddruid.task.executor.service=druid/middleManager -Ddruid.task.executor.host=172.19.0.6 -Ddruid.task.executor.plaintextPort=8091 -Ddruid.task.executor.enablePlaintextPort=true -Ddruid.task.executor.tlsPort=-1 -Ddruid.task.executor.enableTlsPort=false org.apache.druid.cli.Main internal peon var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/task.json var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/e509c961-9133-4e7b-9ed3-a44beb8b5d7b/status.json var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/e509c961-9133-4e7b-9ed3-a44beb8b5d7b/report.json
overlord         | 2019-11-05T22:54:22,106 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote RUNNING status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='null', port=-1, tlsPort=-1}]
middlemanager    | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] location changed to [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}].
middlemanager    | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [RUNNING].
middlemanager    | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z output to: var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/log
overlord         | 2019-11-05T22:54:22,132 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote RUNNING status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}]
overlord         | 2019-11-05T22:54:22,132 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] location changed to [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}].
coordinator      | 2019-11-05T22:54:22,486 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
overlord         | 2019-11-05T22:54:23,391 DEBUG [qtp2107873140-64] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/supervisor?full HTTP/1.1
router           | 2019-11-05T22:54:23,395 DEBUG [qtp883862737-137] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/supervisor?full HTTP/1.1
overlord         | 2019-11-05T22:54:23,407 DEBUG [qtp2107873140-78] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/waitingTasks HTTP/1.1
overlord         | 2019-11-05T22:54:23,407 DEBUG [qtp2107873140-86] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/runningTasks HTTP/1.1
router           | 2019-11-05T22:54:23,410 DEBUG [qtp883862737-143] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/waitingTasks HTTP/1.1
router           | 2019-11-05T22:54:23,411 DEBUG [qtp883862737-142] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/runningTasks HTTP/1.1
router           | 2019-11-05T22:54:23,418 DEBUG [AsyncManagementForwardingServlet-2b1cd7bc-173] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/completeTasks HTTP/1.1
overlord         | 2019-11-05T22:54:23,419 DEBUG [qtp2107873140-72] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/completeTasks HTTP/1.1
overlord         | 2019-11-05T22:54:23,420 DEBUG [qtp2107873140-61] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/pendingTasks HTTP/1.1
router           | 2019-11-05T22:54:23,423 DEBUG [qtp883862737-99] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/pendingTasks HTTP/1.1
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
coordinator      | 2019-11-05T22:54:32,489 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:32,490 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
overlord         | 2019-11-05T22:54:36,182 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced 1 tasks from storage (0 tasks added, 0 tasks removed).
coordinator      | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
coordinator      | 2019-11-05T22:54:39,017 INFO [DatabaseRuleManager-Exec--0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found 1 rule(s) for 1 datasource(s)
coordinator      | 2019-11-05T22:54:40,178 INFO [org.apache.druid.metadata.SQLMetadataSegmentManager-Exec--0] org.apache.druid.metadata.SQLMetadataSegmentManager - No segments found in the database!
coordinator      | 2019-11-05T22:54:42,496 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:42,496 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
coordinator      | 2019-11-05T22:54:44,828 DEBUG [qtp1733520096-108] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.5:8081/druid/coordinator/v1/rules HTTP/1.1
coordinator      | 2019-11-05T22:54:47,498 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:47,498 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
coordinator      | 2019-11-05T22:54:52,500 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:52,500 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:52,502 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
middlemanager    | 2019-11-05T22:54:57,128 INFO [forking-task-runner-1-[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z]] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[1] for task: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z
middlemanager    | 2019-11-05T22:54:57,128 INFO [forking-task-runner-1] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: var/druid/indexing-logs/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z.log
middlemanager    | 2019-11-05T22:54:57,129 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [FAILED].
middlemanager    | 2019-11-05T22:54:57,129 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z
middlemanager    | 2019-11-05T22:54:57,144 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Job's finished. Completed [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] with status [FAILED]
overlord         | 2019-11-05T22:54:57,146 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote FAILED status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}]
overlord         | 2019-11-05T22:54:57,147 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] completed task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] with status[FAILED]
overlord         | 2019-11-05T22:54:57,148 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z
overlord         | 2019-11-05T22:54:57,149 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] because: [notified status change from task]
overlord         | 2019-11-05T22:54:57,149 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on worker[172.19.0.6:8091]
overlord         | 2019-11-05T22:54:57,153 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] from activeTasks
overlord         | 2019-11-05T22:54:57,154 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z to status: TaskStatus{id=index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z, status=FAILED, duration=35039, errorMsg=null}
overlord         | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task done: AbstractTask{id='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', groupId='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', taskResource=TaskResource{availabilityGroup='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', requiredCapacity=1}, dataSource='wikiticker-2015-09-12-sampled', context={forceTimeChunkLock=true}}
overlord         | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: AbstractTask{id='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', groupId='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', taskResource=TaskResource{availabilityGroup='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', requiredCapacity=1}, dataSource='wikiticker-2015-09-12-sampled', context={forceTimeChunkLock=true}} (35039 run duration)
overlord         | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [FAILED].
overlord         | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] went bye bye.
coordinator      | 2019-11-05T22:54:57,503 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:54:57,503 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:54:57,505 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
coordinator      | 2019-11-05T22:55:02,506 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
coordinator      | 2019-11-05T22:55:02,506 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
coordinator      | 2019-11-05T22:55:02,507 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
coordinator      | 2019-11-05T22:55:02,507 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
coordinator      | 2019-11-05T22:55:02,508 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
coordinator      | 2019-11-05T22:55:02,508 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.```
scottbelden commented 4 years ago

It turns out that the main issue was that the default environment file wants to use azure as the storage mechanism:

https://github.com/apache/incubator-druid/blob/dc6178d1f2332fe03e18576713ad517c87d4d031/distribution/docker/environment#L44

I was able to get it working by setting up a docker volume shared by all of the containers and setting the storage mechanism to local with the path set to the shared volume.