Kafka Indexing does not fail ingestion task on OffsetOutOfRangeException even when no reset policy is configured

Full Peon Log:

2022-01-05T11:50:15,791 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final 2022-01-05T11:50:16,272 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.22.1.jar 2022-01-05T11:50:16,273 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.22.1.jar 2022-01-05T11:50:16,275 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-0.22.1.jar, mapdb-1.0.8.jar 2022-01-05T11:50:16,276 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-42.2.14.jar, postgresql-metadata-storage-0.22.1.jar 2022-01-05T11:50:16,276 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-0.22.1.jar, kafka-clients-2.8.0.jar, lz4-java-1.7.1.jar, snappy-java-1.1.8.1.jar, zstd-jni-1.3.3-1.jar 2022-01-05T11:50:16,278 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.22.1.jar 2022-01-05T11:50:16,435 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.22.1.jar 2022-01-05T11:50:16,437 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.22.1.jar 2022-01-05T11:50:16,443 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-0.22.1.jar, mapdb-1.0.8.jar 2022-01-05T11:50:16,445 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-42.2.14.jar, postgresql-metadata-storage-0.22.1.jar 2022-01-05T11:50:16,446 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: druid-kafka-indexing-service-0.22.1.jar, kafka-clients-2.8.0.jar, lz4-java-1.7.1.jar, snappy-java-1.1.8.1.jar, zstd-jni-1.3.3-1.jar 2022-01-05T11:50:16,448 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.22.1.jar 2022-01-05T11:50:17,594 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.http.recipientBaseUrl property: one of it's prefixes is also used as a property key. Prefix: druid 2022-01-05T11:50:17,642 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [HttpPostEmitter{config=HttpEmitterConfig{flushMillis=60000, flushCount=500, flushTimeOut=9223372036854775807, basicAuthentication='null', batchingStrategy=ARRAY, maxBatchSize=5242880, contentEncoding=null, batchQueueSizeLimit=6, httpTimeoutAllowanceFactor=2.0, minHttpTimeoutMillis=0, recipientBaseUrl='http://int-global-prometheus-druid-exporter:8080/druid'}}] for metrics and alerts, with dimensions [{version=0.22.1}]. 2022-01-05T11:50:18,129 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 3 monitors: org.apache.druid.java.util.metrics.JvmMonitor, org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors[2], memory[219,152,384], maxMemory[358,088,704], directMemory[201,326,592]. Properties follow. 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - awt.toolkit: sun.awt.X11.XToolkit 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.authorizerName: MetadataAuthorizer 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.credentialsValidator.type: metadata 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.initialAdminPassword: 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.initialInternalClientPassword: 2022-01-05T11:50:18,133 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.skipOnFailure: false 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticator.MetadataAuthenticator.type: basic 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authenticatorChain: ["MetadataAuthenticator"] 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authorizer.MetadataAuthorizer.type: basic 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.auth.authorizers: ["MetadataAuthorizer"] 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.emitter: http 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.emitter.http.recipientBaseUrl: http://int-global-prometheus-druid-exporter:8080/druid 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.emitter.logging.logLevel: info 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.escalator.authorizerName: MetadataAuthorizer 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.escalator.internalClientPassword: 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.escalator.internalClientUsername: druid_system 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.escalator.type: basic 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.extensions.loadList: ["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-kafka-indexing-service", "druid-basic-security"] 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.generic.useDefaultValueForNull: false 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.global.http.readTimeout: PT1h 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.host: 10.85.6.250 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.fork.property.druid.processing.buffer.sizeBytes: 2500000 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.fork.property.druid.processing.numMergeBuffers: 2 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.fork.property.druid.processing.numThreads: 1 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.logs.directory: /opt/data/indexing-logs 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.logs.type: file 2022-01-05T11:50:18,134 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.runner.javaOpts: -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.runner.javaOptsArray: ["-server", "-Xms128m", "-Xmx384m", "-XX:MaxDirectMemorySize=192m", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"] 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.task.baseTaskDir: var/druid/task 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.indexer.task.hadoopWorkingPath: var/druid/hadoop-tmp 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.indexing.doubleStorage: double 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.lookup.enableLookupSyncOnStartup: false 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.connector.connectURI: jdbc:postgresql://druid-int-postgresql:5432/druid 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.connector.host: localhost 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.connector.password: 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.connector.port: 1527 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.connector.user: druid 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metadata.storage.type: postgresql 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metrics.emitter.dimension.dataSource: DATASOURCE_NAME 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metrics.emitter.dimension.taskId: index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.metrics.emitter.dimension.taskType: index_kafka 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.monitoring.emissionPeriod: PT2m 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.monitoring.monitors: ["org.apache.druid.java.util.metrics.JvmMonitor"] 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.plaintextPort: 8102 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.processing.buffer.sizeBytes: 2500000 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.processing.numMergeBuffers: 2 2022-01-05T11:50:18,135 INFO [main] org.apache.druid.cli.CliPeon - druid.processing.numThreads: 1 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.query.groupBy.maxOnDiskStorage: 500000000 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.selectors.coordinator.serviceName: druid/coordinator 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.selectors.indexing.serviceName: druid/overlord 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.server.hiddenProperties: ["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"] 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.server.http.defaultQueryTimeout: 3600000 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.server.http.maxIdleTime: PT1h 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.server.http.numThreads: 60 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.service: druid/middleManager 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.sql.enable: true 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.startup.logging.logProperties: true 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.storage.storageDirectory: var/druid/segments 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.storage.type: local 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.enablePlaintextPort: true 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.enableTlsPort: false 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.host: 10.85.6.250 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.plaintextPort: 8091 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.service: druid/middleManager 2022-01-05T11:50:18,136 INFO [main] org.apache.druid.cli.CliPeon - druid.task.executor.tlsPort: -1 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - druid.tlsPort: -1 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - druid.worker.capacity: 4 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - druid.zk.paths.base: /druid 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - druid.zk.service.host: zookeeper-int-headless 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - file.encoding: UTF-8 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - file.encoding.pkg: sun.io 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - file.separator: / 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.awt.printerjob: sun.print.PSPrinterJob 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.class.path: /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/FastInfoset-1.2.15.jar:lib/RoaringBitmap-0.9.0.jar:lib/accessors-smart-1.2.jar:lib/aether-api-0.9.0.M2.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/aether-impl-0.9.0.M2.jar:lib/aether-spi-0.9.0.M2.jar:lib/aether-util-0.9.0.M2.jar:lib/aggdesigner-algorithm-6.0.jar:lib/airline-0.7.jar:lib/antlr4-runtime-4.5.1.jar:lib/aopalliance-1.0.jar:lib/asm-7.1.jar:lib/asm-analysis-7.1.jar:lib/asm-commons-7.1.jar:lib/asm-tree-7.1.jar:lib/async-http-client-2.5.3.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/audience-annotations-0.5.0.jar:lib/avatica-core-1.17.0.jar:lib/avatica-metrics-1.17.0.jar:lib/avatica-server-1.17.0.jar:lib/aws-java-sdk-core-1.12.37.jar:lib/aws-java-sdk-ec2-1.12.37.jar:lib/aws-java-sdk-kms-1.12.37.jar:lib/aws-java-sdk-s3-1.12.37.jar:lib/caffeine-2.8.0.jar:lib/calcite-core-1.21.0.jar:lib/calcite-linq4j-1.21.0.jar:lib/checker-qual-2.5.7.jar:lib/classmate-1.1.0.jar:lib/commons-beanutils-1.9.4.jar:lib/commons-codec-1.13.jar:lib/commons-collections-3.2.2.jar:lib/commons-collections4-4.2.jar:lib/commons-compiler-3.0.11.jar:lib/commons-compress-1.21.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-io-2.11.0.jar:lib/commons-lang-2.6.jar:lib/commons-lang3-3.8.1.jar:lib/commons-logging-1.1.1.jar:lib/commons-math3-3.6.1.jar:lib/commons-net-3.6.jar:lib/commons-pool2-2.2.jar:lib/commons-text-1.3.jar:lib/compress-lzf-1.0.4.jar:lib/config-magic-0.9.jar:lib/cron-scheduler-0.1.jar:lib/curator-client-4.3.0.jar:lib/curator-framework-4.3.0.jar:lib/curator-recipes-4.3.0.jar:lib/curator-x-discovery-4.3.0.jar:lib/datasketches-java-2.0.0.jar:lib/datasketches-memory-1.3.0.jar:lib/derby-10.14.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/derbynet-10.14.2.0.jar:lib/disruptor-3.3.6.jar:lib/druid-aws-common-0.22.1.jar:lib/druid-console-0.22.1.jar:lib/druid-core-0.22.1.jar:lib/druid-gcp-common-0.22.1.jar:lib/druid-hll-0.22.1.jar:lib/druid-indexing-hadoop-0.22.1.jar:lib/druid-indexing-service-0.22.1.jar:lib/druid-processing-0.22.1.jar:lib/druid-server-0.22.1.jar:lib/druid-services-0.22.1.jar:lib/druid-sql-0.22.1.jar:lib/error_prone_annotations-2.8.0.jar:lib/esri-geometry-api-2.2.0.jar:lib/extendedset-0.22.1.jar:lib/fastutil-8.5.4.jar:lib/fastutil-core-8.5.4.jar:lib/fastutil-extra-8.5.4.jar:lib/google-api-client-1.26.0.jar:lib/google-http-client-1.26.0.jar:lib/google-http-client-jackson2-1.26.0.jar:lib/google-oauth-client-1.26.0.jar:lib/guava-16.0.1.jar:lib/guice-4.1.0.jar:lib/guice-assistedinject-4.1.0.jar:lib/guice-multibindings-4.1.0.jar:lib/guice-servlet-4.1.0.jar:lib/hibernate-validator-5.2.5.Final.jar:lib/httpclient-4.5.10.jar:lib/httpcore-4.4.11.jar:lib/icu4j-55.1.jar:lib/ion-java-1.0.2.jar:lib/istack-commons-runtime-3.0.7.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotations-2.10.5.jar:lib/jackson-core-2.10.5.jar:lib/jackson-databind-2.10.5.1.jar:lib/jackson-dataformat-cbor-2.10.5.jar:lib/jackson-dataformat-smile-2.10.5.jar:lib/jackson-datatype-guava-2.10.5.jar:lib/jackson-datatype-joda-2.10.5.jar:lib/jackson-jaxrs-base-2.10.5.jar:lib/jackson-jaxrs-json-provider-2.10.5.jar:lib/jackson-jaxrs-smile-provider-2.10.5.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-guice-2.10.5.jar:lib/jackson-module-jaxb-annotations-2.10.5.jar:lib/jakarta.activation-api-1.2.1.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/janino-3.0.11.jar:lib/javax.activation-1.2.0.jar:lib/javax.activation-api-1.2.0.jar:lib/javax.el-3.0.0.jar:lib/javax.el-api-3.0.0.jar:lib/javax.inject-1.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-api-2.3.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/jboss-logging-3.2.1.Final.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/jcodings-1.0.43.jar:lib/jdbi-2.63.1.jar:lib/jersey-core-1.19.3.jar:lib/jersey-guice-1.19.3.jar:lib/jersey-server-1.19.3.jar:lib/jersey-servlet-1.19.3.jar:lib/jetty-client-9.4.40.v20210413.jar:lib/jetty-continuation-9.4.40.v20210413.jar:lib/jetty-http-9.4.40.v20210413.jar:lib/jetty-io-9.4.40.v20210413.jar:lib/jetty-proxy-9.4.40.v20210413.jar:lib/jetty-rewrite-9.4.40.v20210413.jar:lib/jetty-security-9.4.40.v20210413.jar:lib/jetty-server-9.4.40.v20210413.jar:lib/jetty-servlet-9.4.40.v20210413.jar:lib/jetty-servlets-9.4.40.v20210413.jar:lib/jetty-util-9.4.40.v20210413.jar:lib/jetty-util-ajax-9.4.40.v20210413.jar:lib/jmespath-java-1.12.37.jar:lib/jna-4.5.1.jar:lib/joda-time-2.10.5.jar:lib/joni-2.1.27.jar:lib/json-path-2.3.0.jar:lib/json-smart-2.3.jar:lib/jsr305-2.0.1.jar:lib/jsr311-api-1.1.1.jar:lib/jvm-attach-api-1.5.jar:lib/log4j-1.2-api-2.15.0.jar:lib/log4j-api-2.15.0.jar:lib/log4j-core-2.15.0.jar:lib/log4j-jul-2.15.0.jar:lib/log4j-slf4j-impl-2.15.0.jar:lib/lz4-java-1.7.1.jar:lib/maven-aether-provider-3.1.1.jar:lib/maven-artifact-3.6.0.jar:lib/maven-model-3.1.1.jar:lib/maven-model-builder-3.1.1.jar:lib/maven-repository-metadata-3.1.1.jar:lib/maven-settings-3.1.1.jar:lib/maven-settings-builder-3.1.1.jar:lib/metrics-core-4.0.0.jar:lib/netty-3.10.6.Final.jar:lib/netty-buffer-4.1.68.Final.jar:lib/netty-codec-4.1.68.Final.jar:lib/netty-codec-dns-4.1.68.Final.jar:lib/netty-codec-http-4.1.68.Final.jar:lib/netty-codec-socks-4.1.68.Final.jar:lib/netty-common-4.1.68.Final.jar:lib/netty-handler-4.1.68.Final.jar:lib/netty-handler-proxy-4.1.68.Final.jar:lib/netty-reactive-streams-2.0.0.jar:lib/netty-resolver-4.1.68.Final.jar:lib/netty-resolver-dns-4.1.68.Final.jar:lib/netty-transport-4.1.68.Final.jar:lib/netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/netty-transport-native-unix-common-4.1.68.Final.jar:lib/okhttp-1.0.2.jar:lib/opencsv-4.6.jar:lib/plexus-interpolation-1.19.jar:lib/plexus-utils-3.0.24.jar:lib/protobuf-java-3.11.0.jar:lib/reactive-streams-1.0.2.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/resilience4j-core-1.3.1.jar:lib/rhino-1.7.11.jar:lib/shims-0.9.0.jar:lib/sigar-1.6.5.132.jar:lib/slf4j-api-1.7.25.jar:lib/spymemcached-2.12.3.jar:lib/stax-ex-1.8.jar:lib/tesla-aether-0.0.5.jar:lib/txw2-2.3.1.jar:lib/validation-api-1.1.0.Final.jar:lib/vavr-0.10.2.jar:lib/vavr-match-0.10.2.jar:lib/xz-1.8.jar:lib/wagon-provider-api-2.4.jar:lib/zookeeper-3.5.9.jar:lib/zookeeper-jute-3.5.9.jar:lib/zstd-jni-1.3.3-1.jar: 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.class.version: 52.0 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.endorsed.dirs: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/endorsed 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.ext.dirs: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/ext:/usr/java/packages/lib/ext 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.io.tmpdir: var/tmp 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.runtime.name: OpenJDK Runtime Environment 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.runtime.version: 1.8.0_275-b01 2022-01-05T11:50:18,137 INFO [main] org.apache.druid.cli.CliPeon - java.specification.name: Java Platform API Specification 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.specification.vendor: Oracle Corporation 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.specification.version: 1.8 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.util.logging.manager: org.apache.logging.log4j.jul.LogManager 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vendor: Red Hat, Inc. 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vendor.url: https://www.redhat.com/ 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vendor.url.bug: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-1.8.0-openjdk 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.version: 1.8.0_275 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.info: mixed mode 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.name: OpenJDK 64-Bit Server VM 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.specification.name: Java Virtual Machine Specification 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.specification.vendor: Oracle Corporation 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.specification.version: 1.8 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.vendor: Red Hat, Inc. 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - java.vm.version: 25.275-b01 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - * line.separator:

2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - log4j.shutdownHookEnabled: true 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - log4j2.is.webapp: false 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - os.arch: amd64 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - os.name: Linux 2022-01-05T11:50:18,138 INFO [main] org.apache.druid.cli.CliPeon - os.version: 4.18.0-305.28.1.el8_4.x86_64 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - path.separator: : 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.arch.data.model: 64 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.boot.class.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/classes 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.boot.library.path: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre/lib/amd64 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.cpu.endian: little 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.cpu.isalist: 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.io.unicode.encoding: UnicodeLittle 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.java.command: org.apache.druid.cli.Main internal peon var/druid/task/index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao/task.json var/druid/task/index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao/d3bb8b64-f303-4e1b-8490-31c18561577a/status.json var/druid/task/index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao/d3bb8b64-f303-4e1b-8490-31c18561577a/report.json --loadBroadcastSegments true 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.java.launcher: SUN_STANDARD 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.jnu.encoding: ANSI_X3.4-1968 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.management.compiler: HotSpot 64-Bit Tiered Compilers 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - sun.os.patch.level: unknown 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.country: US 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.dir: /opt/druid 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.home: ? 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.language: en 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.name: ? 2022-01-05T11:50:18,139 INFO [main] org.apache.druid.cli.CliPeon - user.timezone: UTC 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid-int-middle-manager-2.druid-int-middle-manager.some_proj_name-global-int.svc.cluster.local 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_275 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Red Hat, Inc. 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-1.el8_3.x86_64/jre 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/FastInfoset-1.2.15.jar:lib/RoaringBitmap-0.9.0.jar:lib/accessors-smart-1.2.jar:lib/aether-api-0.9.0.M2.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/aether-impl-0.9.0.M2.jar:lib/aether-spi-0.9.0.M2.jar:lib/aether-util-0.9.0.M2.jar:lib/aggdesigner-algorithm-6.0.jar:lib/airline-0.7.jar:lib/antlr4-runtime-4.5.1.jar:lib/aopalliance-1.0.jar:lib/asm-7.1.jar:lib/asm-analysis-7.1.jar:lib/asm-commons-7.1.jar:lib/asm-tree-7.1.jar:lib/async-http-client-2.5.3.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/audience-annotations-0.5.0.jar:lib/avatica-core-1.17.0.jar:lib/avatica-metrics-1.17.0.jar:lib/avatica-server-1.17.0.jar:lib/aws-java-sdk-core-1.12.37.jar:lib/aws-java-sdk-ec2-1.12.37.jar:lib/aws-java-sdk-kms-1.12.37.jar:lib/aws-java-sdk-s3-1.12.37.jar:lib/caffeine-2.8.0.jar:lib/calcite-core-1.21.0.jar:lib/calcite-linq4j-1.21.0.jar:lib/checker-qual-2.5.7.jar:lib/classmate-1.1.0.jar:lib/commons-beanutils-1.9.4.jar:lib/commons-codec-1.13.jar:lib/commons-collections-3.2.2.jar:lib/commons-collections4-4.2.jar:lib/commons-compiler-3.0.11.jar:lib/commons-compress-1.21.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-io-2.11.0.jar:lib/commons-lang-2.6.jar:lib/commons-lang3-3.8.1.jar:lib/commons-logging-1.1.1.jar:lib/commons-math3-3.6.1.jar:lib/commons-net-3.6.jar:lib/commons-pool2-2.2.jar:lib/commons-text-1.3.jar:lib/compress-lzf-1.0.4.jar:lib/config-magic-0.9.jar:lib/cron-scheduler-0.1.jar:lib/curator-client-4.3.0.jar:lib/curator-framework-4.3.0.jar:lib/curator-recipes-4.3.0.jar:lib/curator-x-discovery-4.3.0.jar:lib/datasketches-java-2.0.0.jar:lib/datasketches-memory-1.3.0.jar:lib/derby-10.14.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/derbynet-10.14.2.0.jar:lib/disruptor-3.3.6.jar:lib/druid-aws-common-0.22.1.jar:lib/druid-console-0.22.1.jar:lib/druid-core-0.22.1.jar:lib/druid-gcp-common-0.22.1.jar:lib/druid-hll-0.22.1.jar:lib/druid-indexing-hadoop-0.22.1.jar:lib/druid-indexing-service-0.22.1.jar:lib/druid-processing-0.22.1.jar:lib/druid-server-0.22.1.jar:lib/druid-services-0.22.1.jar:lib/druid-sql-0.22.1.jar:lib/error_prone_annotations-2.8.0.jar:lib/esri-geometry-api-2.2.0.jar:lib/extendedset-0.22.1.jar:lib/fastutil-8.5.4.jar:lib/fastutil-core-8.5.4.jar:lib/fastutil-extra-8.5.4.jar:lib/google-api-client-1.26.0.jar:lib/google-http-client-1.26.0.jar:lib/google-http-client-jackson2-1.26.0.jar:lib/google-oauth-client-1.26.0.jar:lib/guava-16.0.1.jar:lib/guice-4.1.0.jar:lib/guice-assistedinject-4.1.0.jar:lib/guice-multibindings-4.1.0.jar:lib/guice-servlet-4.1.0.jar:lib/hibernate-validator-5.2.5.Final.jar:lib/httpclient-4.5.10.jar:lib/httpcore-4.4.11.jar:lib/icu4j-55.1.jar:lib/ion-java-1.0.2.jar:lib/istack-commons-runtime-3.0.7.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotations-2.10.5.jar:lib/jackson-core-2.10.5.jar:lib/jackson-databind-2.10.5.1.jar:lib/jackson-dataformat-cbor-2.10.5.jar:lib/jackson-dataformat-smile-2.10.5.jar:lib/jackson-datatype-guava-2.10.5.jar:lib/jackson-datatype-joda-2.10.5.jar:lib/jackson-jaxrs-base-2.10.5.jar:lib/jackson-jaxrs-json-provider-2.10.5.jar:lib/jackson-jaxrs-smile-provider-2.10.5.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-guice-2.10.5.jar:lib/jackson-module-jaxb-annotations-2.10.5.jar:lib/jakarta.activation-api-1.2.1.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/janino-3.0.11.jar:lib/javax.activation-1.2.0.jar:lib/javax.activation-api-1.2.0.jar:lib/javax.el-3.0.0.jar:lib/javax.el-api-3.0.0.jar:lib/javax.inject-1.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-api-2.3.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/jboss-logging-3.2.1.Final.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/jcodings-1.0.43.jar:lib/jdbi-2.63.1.jar:lib/jersey-core-1.19.3.jar:lib/jersey-guice-1.19.3.jar:lib/jersey-server-1.19.3.jar:lib/jersey-servlet-1.19.3.jar:lib/jetty-client-9.4.40.v20210413.jar:lib/jetty-continuation-9.4.40.v20210413.jar:lib/jetty-http-9.4.40.v20210413.jar:lib/jetty-io-9.4.40.v20210413.jar:lib/jetty-proxy-9.4.40.v20210413.jar:lib/jetty-rewrite-9.4.40.v20210413.jar:lib/jetty-security-9.4.40.v20210413.jar:lib/jetty-server-9.4.40.v20210413.jar:lib/jetty-servlet-9.4.40.v20210413.jar:lib/jetty-servlets-9.4.40.v20210413.jar:lib/jetty-util-9.4.40.v20210413.jar:lib/jetty-util-ajax-9.4.40.v20210413.jar:lib/jmespath-java-1.12.37.jar:lib/jna-4.5.1.jar:lib/joda-time-2.10.5.jar:lib/joni-2.1.27.jar:lib/json-path-2.3.0.jar:lib/json-smart-2.3.jar:lib/jsr305-2.0.1.jar:lib/jsr311-api-1.1.1.jar:lib/jvm-attach-api-1.5.jar:lib/log4j-1.2-api-2.15.0.jar:lib/log4j-api-2.15.0.jar:lib/log4j-core-2.15.0.jar:lib/log4j-jul-2.15.0.jar:lib/log4j-slf4j-impl-2.15.0.jar:lib/lz4-java-1.7.1.jar:lib/maven-aether-provider-3.1.1.jar:lib/maven-artifact-3.6.0.jar:lib/maven-model-3.1.1.jar:lib/maven-model-builder-3.1.1.jar:lib/maven-repository-metadata-3.1.1.jar:lib/maven-settings-3.1.1.jar:lib/maven-settings-builder-3.1.1.jar:lib/metrics-core-4.0.0.jar:lib/netty-3.10.6.Final.jar:lib/netty-buffer-4.1.68.Final.jar:lib/netty-codec-4.1.68.Final.jar:lib/netty-codec-dns-4.1.68.Final.jar:lib/netty-codec-http-4.1.68.Final.jar:lib/netty-codec-socks-4.1.68.Final.jar:lib/netty-common-4.1.68.Final.jar:lib/netty-handler-4.1.68.Final.jar:lib/netty-handler-proxy-4.1.68.Final.jar:lib/netty-reactive-streams-2.0.0.jar:lib/netty-resolver-4.1.68.Final.jar:lib/netty-resolver-dns-4.1.68.Final.jar:lib/netty-transport-4.1.68.Final.jar:lib/netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/netty-transport-native-unix-common-4.1.68.Final.jar:lib/okhttp-1.0.2.jar:lib/opencsv-4.6.jar:lib/plexus-interpolation-1.19.jar:lib/plexus-utils-3.0.24.jar:lib/protobuf-java-3.11.0.jar:lib/reactive-streams-1.0.2.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/resilience4j-core-1.3.1.jar:lib/rhino-1.7.11.jar:lib/shims-0.9.0.jar:lib/sigar-1.6.5.132.jar:lib/slf4j-api-1.7.25.jar:lib/spymemcached-2.12.3.jar:lib/stax-ex-1.8.jar:lib/tesla-aether-0.0.5.jar:lib/txw2-2.3.1.jar:lib/validation-api-1.1.0.Final.jar:lib/vavr-0.10.2.jar:lib/vavr-match-0.10.2.jar:lib/xz-1.8.jar:lib/wagon-provider-api-2.4.jar:lib/zookeeper-3.5.9.jar:lib/zookeeper-jute-3.5.9.jar:lib/zstd-jni-1.3.3-1.jar: 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=var/tmp 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler= 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2022-01-05T11:50:18,511 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.18.0-305.28.1.el8_4.x86_64 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=? 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=? 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/druid 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=127MB 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=341MB 2022-01-05T11:50:18,512 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=217MB 2022-01-05T11:50:18,512 INFO [main] org.apache.curator.utils.Compatibility - Using emulated InjectSessionExpiration 2022-01-05T11:50:18,552 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS] 2022-01-05T11:50:18,552 INFO [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode:[CLOSED_SEGMENTS] 2022-01-05T11:50:18,653 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy] 2022-01-05T11:50:18,708 INFO [main] org.apache.druid.guice.StorageNodeModule - Segment cache not configured on ServerType [indexer-executor]. It will not be assignable for segment placement 2022-01-05T11:50:18,768 INFO [main] org.eclipse.jetty.util.log - Logging initialized @3742ms to org.eclipse.jetty.util.log.Slf4jLog 2022-01-05T11:50:18,781 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8102] 2022-01-05T11:50:18,932 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23 2022-01-05T11:50:18,961 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[2,500,000] 2022-01-05T11:50:19,036 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[2,500,000] 2022-01-05T11:50:19,037 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[2,500,000] 2022-01-05T11:50:19,047 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [INIT] 2022-01-05T11:50:19,047 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [NORMAL] 2022-01-05T11:50:19,051 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2022-01-05T11:50:19,056 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper-int-headless sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@7185e611 2022-01-05T11:50:19,108 INFO [main] org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2022-01-05T11:50:19,112 INFO [main] org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes 2022-01-05T11:50:19,120 INFO [main] org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled= 2022-01-05T11:50:19,127 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema 2022-01-05T11:50:19,129 INFO [main-SendThread(zookeeper-int-headless:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper-int-headless/10.84.7.11:2181. Will not attempt to authenticate using SASL (unknown error) 2022-01-05T11:50:19,135 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - Starting CoordinatorPollingBasicAuthenticatorCacheManager. 2022-01-05T11:50:19,136 INFO [main-SendThread(zookeeper-int-headless:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.85.6.250:46610, server: zookeeper-int-headless/10.84.7.11:2181 2022-01-05T11:50:19,145 INFO [main-SendThread(zookeeper-int-headless:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper-int-headless/10.84.7.11:2181, sessionid = 0x20006c33aae8561, negotiated timeout = 30000 2022-01-05T11:50:19,148 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2022-01-05T11:50:19,208 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {server.1=zookeeper-int-0.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181, version=0, server.3=zookeeper-int-2.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181, server.2=zookeeper-int-1.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181} 2022-01-05T11:50:19,216 WARN [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type[CONNECTION_RECONNECTED] for node watcher of role[coordinator]. 2022-01-05T11:50:19,221 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {server.1=zookeeper-int-0.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181, version=0, server.3=zookeeper-int-2.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181, server.2=zookeeper-int-1.zookeeper-int-headless.some_proj_name-global-int.svc.cluster.local:2888:3888:participant;0.0.0.0:2181} 2022-01-05T11:50:19,238 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node[http://10.85.5.178:8081] of role[coordinator] detected. 2022-01-05T11:50:19,238 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role[coordinator] is now initialized. 2022-01-05T11:50:19,545 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - Started CoordinatorPollingBasicAuthenticatorCacheManager. 2022-01-05T11:50:19,545 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - Starting CoordinatorPollingBasicAuthorizerCacheManager. 2022-01-05T11:50:19,663 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - Started CoordinatorPollingBasicAuthorizerCacheManager. 2022-01-05T11:50:19,675 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node[http://10.85.5.178:8081] of role[overlord] detected. 2022-01-05T11:50:19,710 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role[overlord] is now initialized. 2022-01-05T11:50:19,814 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: { "type" : "index_kafka", "id" : "index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao", "resource" : { "availabilityGroup" : "index_kafka_DATASOURCE_NAME_11fc40b96a318a6", "requiredCapacity" : 1 }, "dataSchema" : { "dataSource" : "DATASOURCE_NAME", "timestampSpec" : { "column" : "CREATEDAT", "format" : "auto", "missingValue" : null }, "dimensionsSpec" : { "dimensions" : [ ommited ], "dimensionExclusions" : [ "__time" ] }, "metricsSpec" : [ ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : { "type" : "none" }, "rollup" : false, "intervals" : [ ] }, "transformSpec" : { "filter" : null, "transforms" : [ ] } }, "tuningConfig" : { "type" : "KafkaTuningConfig", "appendableIndexSpec" : { "type" : "onheap" }, "maxRowsInMemory" : 1000000, "maxBytesInMemory" : 0, "skipBytesInMemoryOverheadCheck" : false, "maxRowsPerSegment" : 5000000, "maxTotalRows" : null, "intermediatePersistPeriod" : "PT60S", "basePersistDirectory" : "/opt/druid/var/tmp/druid-realtime-persist1014422271737391165", "maxPendingPersists" : 0, "indexSpec" : { "bitmap" : { "type" : "roaring", "compressRunOnSerialization" : true }, "dimensionCompression" : "lz4", "metricCompression" : "lz4", "longEncoding" : "longs", "segmentLoader" : null }, "indexSpecForIntermediatePersists" : { "bitmap" : { "type" : "roaring", "compressRunOnSerialization" : true }, "dimensionCompression" : "lz4", "metricCompression" : "lz4", "longEncoding" : "longs", "segmentLoader" : null }, "reportParseExceptions" : false, "handoffConditionTimeout" : 0, "resetOffsetAutomatically" : false, "segmentWriteOutMediumFactory" : null, "intermediateHandoffPeriod" : "P2147483647D", "logParseExceptions" : true, "maxParseExceptions" : 2147483647, "maxSavedParseExceptions" : 100, "skipSequenceNumberAvailabilityCheck" : false }, "ioConfig" : { "type" : "kafka", "taskGroupId" : 0, "baseSequenceName" : "index_kafka_DATASOURCE_NAME_11fc40b96a318a6", "startPartitions" : { "type" : "end", "stream" : "DATASOURCE_NAME", "topic" : "DATASOURCE_NAME", "partitionSequenceNumberMap" : { "0" : 39476 }, "partitionOffsetMap" : { "0" : 39476 } }, "endPartitions" : { "type" : "end", "stream" : "DATASOURCE_NAME", "topic" : "DATASOURCE_NAME", "partitionSequenceNumberMap" : { "0" : 9223372036854775807 }, "partitionOffsetMap" : { "0" : 9223372036854775807 } }, "startSequenceNumbers" : { "type" : "start", "stream" : "DATASOURCE_NAME", "topic" : "DATASOURCE_NAME", "partitionSequenceNumberMap" : { "0" : 39476 }, "partitionOffsetMap" : { "0" : 39476 }, "exclusivePartitions" : [ ] }, "endSequenceNumbers" : { "type" : "end", "stream" : "DATASOURCE_NAME", "topic" : "DATASOURCE_NAME", "partitionSequenceNumberMap" : { "0" : 9223372036854775807 }, "partitionOffsetMap" : { "0" : 9223372036854775807 } }, "consumerProperties" : { "bootstrap.servers" : "kafka-int-kafka-bootstrap.some_proj_name-global-int.svc.cluster.local:9093", "security.protocol" : "SSL", "ssl.truststore.location" : "/pki/ca/ca.p12", "ssl.truststore.password" : { "type" : "environment", "variable" : "KAFKA_CLUSTER_CA_PASSWORD" }, "ssl.keystore.location" : "/pki/user/user.p12", "ssl.keystore.password" : { "type" : "environment", "variable" : "default_user" }, "ssl.key.password" : { "type" : "environment", "variable" : "default_user" } }, "pollTimeout" : 100, "useTransaction" : true, "minimumMessageTime" : null, "maximumMessageTime" : null, "inputFormat" : { "type" : "json", "flattenSpec" : { "useFieldDiscovery" : true, "fields" : [ ] }, "featureSpec" : { } } }, "context" : { "checkpoints" : "{\"0\":{\"0\":39476}}", "useLineageBasedSegmentAllocation" : true, "IS_INCREMENTAL_HANDOFF_SUPPORTED" : true, "forceTimeChunkLock" : true }, "groupId" : "index_kafka_DATASOURCE_NAME", "dataSource" : "DATASOURCE_NAME" } 2022-01-05T11:50:19,815 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[var/druid/task/index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao/lock]. 2022-01-05T11:50:19,817 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[var/druid/task/index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao/lock] in 1ms. 2022-01-05T11:50:19,824 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy] 2022-01-05T11:50:19,830 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Running task: index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao 2022-01-05T11:50:19,831 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Starting... 2022-01-05T11:50:19,831 INFO [main] org.apache.druid.server.coordination.CuratorDataSegmentServerAnnouncer - Announcing self[DruidServerMetadata{name='10.85.6.250:8102', hostAndPort='10.85.6.250:8102', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}] at [/druid/announcements/10.85.6.250:8102] 2022-01-05T11:50:19,834 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Started. 2022-01-05T11:50:19,834 INFO [main] org.apache.druid.server.coordination.ZkCoordinator - Starting zkCoordinator for server[10.85.6.250:8102] 2022-01-05T11:50:19,902 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [SERVER] 2022-01-05T11:50:19,906 INFO [main] org.eclipse.jetty.server.Server - jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_275-b01 2022-01-05T11:50:19,917 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Starting with sequences: [SequenceMetadata{sequenceId=0, sequenceName='index_kafka_DATASOURCE_NAME_11fc40b96a318a6_0', assignments=[0], startOffsets={0=39476}, exclusiveStartPartitions=[], endOffsets={0=9223372036854775807}, sentinel=false, checkpointed=false}] 2022-01-05T11:50:20,005 INFO [main] org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0 2022-01-05T11:50:20,005 INFO [main] org.eclipse.jetty.server.session - No SessionScavenger set, using defaults 2022-01-05T11:50:20,006 INFO [main] org.eclipse.jetty.server.session - node0 Scavenging every 600000ms 2022-01-05T11:50:20,028 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.ConsumerConfig - ConsumerConfig values: allow.auto.create.topics = true auto.commit.interval.ms = 5000 auto.offset.reset = none bootstrap.servers = [kafka-int-kafka-bootstrap.some_proj_name-global-int.svc.cluster.local:9093] check.crcs = true client.dns.lookup = use_all_dns_ips client.id = consumer-kafka-supervisor-knonnadg-1 client.rack = connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = kafka-supervisor-knonnadg group.instance.id = null heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true internal.throw.on.fetch.stable.offset.unsupported = false isolation.level = read_committed key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 10000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = SSL security.providers = null send.buffer.bytes = 131072 session.timeout.ms = 10000 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = [hidden] ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = /pki/user/user.p12 ssl.keystore.password = [hidden] ssl.keystore.type = JKS ssl.protocol = TLSv1.2 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = /pki/ca/ca.p12 ssl.truststore.password = [hidden] ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

2022-01-05T11:50:20,128 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43 PM' 2022-01-05T11:50:20,820 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.8.0 2022-01-05T11:50:20,820 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: ebb1d6e21cc92130 2022-01-05T11:50:20,820 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1641383420818 2022-01-05T11:50:20,830 INFO [task-runner-0-priority-0] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Announced self [{"druidNode":{"service":"druid/middleManager","host":"10.85.6.250","bindOnHost":false,"plaintextPort":8102,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"peon","services":{"dataNodeService":{"type":"dataNodeService","tier":"_default_tier","maxSize":0,"type":"indexer-executor","priority":0},"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}]. 2022-01-05T11:50:20,930 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Initialized sequences: SequenceMetadata{sequenceId=0, sequenceName='index_kafka_DATASOURCE_NAME_11fc40b96a318a6_0', assignments=[0], startOffsets={0=39476}, exclusiveStartPartitions=[], endOffsets={0=9223372036854775807}, sentinel=false, checkpointed=false} 2022-01-05T11:50:20,931 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Adding partition[0], start[39476] -> end[9223372036854775807] to assignment. 2022-01-05T11:50:20,933 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Subscribed to partition(s): DATASOURCE_NAME-0 2022-01-05T11:50:20,937 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Seeking partition[0] to[39476]. 2022-01-05T11:50:20,937 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Seeking to offset 39476 for partition DATASOURCE_NAME-0 2022-01-05T11:50:21,411 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@38826984{/,null,AVAILABLE} 2022-01-05T11:50:21,421 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@3f1d6a13{HTTP/1.1, (http/1.1)}{0.0.0.0:8102} 2022-01-05T11:50:21,421 INFO [main] org.eclipse.jetty.server.Server - Started @6397ms 2022-01-05T11:50:21,422 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS] 2022-01-05T11:50:21,445 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module] 2022-01-05T11:50:22,245 INFO [task-runner-0-priority-0] org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Cluster ID: LCOjcmu5S4Cfusr5J4ji5A 2022-01-05T11:50:22,403 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:50:22,404 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:50:22,404 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:50:52,407 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:50:52,407 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:50:52,408 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:51:22,411 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:51:22,411 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:51:22,411 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:51:52,414 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:51:52,414 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:51:52,414 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:52:22,418 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:52:22,418 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:52:22,418 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:52:52,421 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0, raising error to the application since no reset policy is configured 2022-01-05T11:52:52,421 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=39476, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-int-kafka-2.kafka-int-kafka-brokers.some_proj_name-global-int.svc:9093 (id: 2 rack: null)], epoch=8}} is out of range for partition DATASOURCE_NAME-0] 2022-01-05T11:52:52,421 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms 2022-01-05T11:53:20,937 INFO [qtp1010040056-126] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Sequence[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_0] end offsets updated from [{0=9223372036854775807}] to [{0=39476}]. 2022-01-05T11:53:20,944 INFO [qtp1010040056-126] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Saved sequence metadata to disk: [SequenceMetadata{sequenceId=0, sequenceName='index_kafka_DATASOURCE_NAME_11fc40b96a318a6_0', assignments=[0], startOffsets={0=39476}, exclusiveStartPartitions=[], endOffsets={0=39476}, sentinel=false, checkpointed=true}] 2022-01-05T11:53:20,944 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Finished reading partition[0], up to[39476]. 2022-01-05T11:53:20,944 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.KafkaConsumer - [Consumer clientId=consumer-kafka-supervisor-knonnadg-1, groupId=kafka-supervisor-knonnadg] Unsubscribed all topics or patterns and assigned partitions 2022-01-05T11:53:20,949 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted rows[0] and (estimated) bytes[0] 2022-01-05T11:53:20,955 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data with commit metadata [AppenderatorDriverMetadata{segments={}, lastSegmentIds={}, callerMetadata={nextPartitions=SeekableStreamEndSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}}}}] for segments: 2022-01-05T11:53:20,955 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted stats: processed rows: [0], persisted rows[0], sinks: [0], total fireHydrants (across sinks): [0], persisted fireHydrants (across sinks): [0] 2022-01-05T11:53:20,958 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing [0] segments in background 2022-01-05T11:53:20,960 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted rows[0] and (estimated) bytes[0] 2022-01-05T11:53:20,967 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Flushed in-memory data with commit metadata [AppenderatorDriverMetadata{segments={}, lastSegmentIds={}, callerMetadata={nextPartitions=SeekableStreamStartSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}, exclusivePartitions=[]}, publishPartitions=SeekableStreamEndSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}}}}] for segments: 2022-01-05T11:53:20,968 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted stats: processed rows: [0], persisted rows[0], sinks: [0], total fireHydrants (across sinks): [0], persisted fireHydrants (across sinks): [0] 2022-01-05T11:53:20,968 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Preparing to push (stats): processed rows: [0], sinks: [0], fireHydrants (across sinks): [0] 2022-01-05T11:53:20,969 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Push complete... 2022-01-05T11:53:20,975 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-publish] org.apache.druid.indexing.seekablestream.SequenceMetadata - With empty segment set, start offsets [SeekableStreamStartSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}, exclusivePartitions=[]}] and end offsets [SeekableStreamEndSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}}] are the same, skipping metadata commit. 2022-01-05T11:53:20,976 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-publish] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Published [0] segments with commit metadata [{nextPartitions=SeekableStreamStartSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}, exclusivePartitions=[]}, publishPartitions=SeekableStreamEndSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}}}] 2022-01-05T11:53:20,977 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-publish] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Published 0 segments for sequence [index_kafka_DATASOURCE_NAME_11fc40b96a318a6_0] with metadata [AppenderatorDriverMetadata{segments={}, lastSegmentIds={}, callerMetadata={nextPartitions=SeekableStreamStartSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}, exclusivePartitions=[]}, publishPartitions=SeekableStreamEndSequenceNumbers{stream='DATASOURCE_NAME', partitionSequenceNumberMap={0=39476}}}}]. 2022-01-05T11:53:20,980 INFO [[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]-publish] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Saved sequence metadata to disk: [] 2022-01-05T11:53:20,982 INFO [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Handoff complete for segments: 2022-01-05T11:53:20,984 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed 2022-01-05T11:53:20,984 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter 2022-01-05T11:53:20,984 INFO [task-runner-0-priority-0] org.apache.kafka.common.metrics.Metrics - Metrics reporters closed 2022-01-05T11:53:20,988 INFO [task-runner-0-priority-0] org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-kafka-supervisor-knonnadg-1 unregistered 2022-01-05T11:53:20,989 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/PEON/10.85.6.250:8102] 2022-01-05T11:53:21,002 INFO [task-runner-0-priority-0] org.apache.druid.curator.discovery.CuratorDruidNodeAnnouncer - Unannounced self [{"druidNode":{"service":"druid/middleManager","host":"10.85.6.250","bindOnHost":false,"plaintextPort":8102,"port":-1,"tlsPort":-1,"enablePlaintextPort":true,"enableTlsPort":false},"nodeType":"peon","services":{"dataNodeService":{"type":"dataNodeService","tier":"_default_tier","maxSize":0,"type":"indexer-executor","priority":0},"lookupNodeService":{"type":"lookupNodeService","lookupTier":"__default"}}}]. 2022-01-05T11:53:21,002 INFO [task-runner-0-priority-0] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/10.85.6.250:8102] 2022-01-05T11:53:21,014 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao", "status" : "SUCCESS", "duration" : 181183, "errorMsg" : null, "location" : { "host" : null, "port" : -1, "tlsPort" : -1 } } 2022-01-05T11:53:21,021 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS] 2022-01-05T11:53:21,026 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER] 2022-01-05T11:53:21,031 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@3f1d6a13{HTTP/1.1, (http/1.1)}{0.0.0.0:8102} 2022-01-05T11:53:21,031 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging 2022-01-05T11:53:21,032 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@38826984{/,null,STOPPED} 2022-01-05T11:53:21,036 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL] 2022-01-05T11:53:21,036 INFO [main] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='10.85.6.250:8102', hostAndPort='10.85.6.250:8102', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}] 2022-01-05T11:53:21,036 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping... 2022-01-05T11:53:21,036 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped. 2022-01-05T11:53:21,039 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_DATASOURCE_NAME_11fc40b96a318a6_pagmfbao]. 2022-01-05T11:53:21,039 INFO [main] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [PUBLISHING]) 2022-01-05T11:53:21,053 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup [definitionkey_to_fachverfahren]. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup [batchstate_to_detailstate]. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup [batchstate_to_description]. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup [batchstate_to_overviewstate]. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.query.lookup.LookupReferencesManager - Closed lookup [definitionkey_to_programmname]. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping. 2022-01-05T11:53:21,054 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped. 2022-01-05T11:53:21,065 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 2022-01-05T11:53:21,173 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x20006c33aae8561 closed 2022-01-05T11:53:21,173 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x20006c33aae8561 2022-01-05T11:53:21,202 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$CloseableHandler - Closing object[org.asynchttpclient.DefaultAsyncHttpClient@49cceb19] 2022-01-05T11:53:21,203 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT] Finished peon task

apache / druid

Kafka Indexing does not fail ingestion task on OffsetOutOfRangeException even when no reset policy is configured #12123

Affected Version

Description