search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.32k stars 3.66k forks source link

druid 0.11.0 kafka-emitter extension error #5198

Closed quenlang closed 6 years ago

quenlang commented 6 years ago

I already uploaded kafka-emitter jars to the extensions directory and loaded the extension.

[root@gpsegment1 druid-0.11.0]# ls extensions/kafka-emitter/
kafka-clients-0.10.2.0.jar  kafka-emitter-0.11.0.jar  lz4-1.3.0.jar  slf4j-api-1.6.4.jar  snappy-java-1.1.2.6.jar

[root@gpsegment1 druid-0.11.0]# grep "kafka-emitter" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties 
druid.extensions.loadList=["druid-hdfs-storage", "kafka-emitter", "druid-caffeine-cache", "druid-kafka-indexing-service", "mysql-metadata-storage", "druid-datasketches", "druid-histogram"]

[root@gpsegment1 druid-0.11.0]# grep "emitter.kafka" /opt/druid-0.11.0/conf/druid/_common/common.runtime.properties 
druid.emitter=kafka
druid.emitter.logging.logLevel=info
druid.emitter.kafka.bootstrap.servers=192.168.1.115:9092
druid.emitter.kafka.metric.topic=druid-metrics-monitor
druid.emitter.kafka.alert.topic=druid-metrics-alert
druid.emitter.kafka.producer.config={"max.block.ms":10000}

When i start the service ,error occur.but under version 0.10.0, kafka-emitter works normally.my kafka version is 0.10.2. Is this a bug?

2017-12-27T15:55:43,216 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@20a7953c] 2017-12-27T15:55:43,337 INFO [main] org.apache.curator.utils.Compatibility - Running in ZooKeeper 3.4.x compatibility mode 2017-12-27T15:55:43,340 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29 2017-12-27T15:55:43,416 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d] 2017-12-27T15:55:43,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.security.Escalator] from props[druid.escalator.] as [io.druid.server.security.NoopEscalator@78422efb] 2017-12-27T15:55:43,480 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{authenticatorChain='null', authorizers='null'}] Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Problem parsing object at prefix[druid.emitter.kafka]: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null. at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule) at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:133) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule) while locating com.google.common.base.Supplier at io.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:134) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule) while locating io.druid.emitter.kafka.KafkaEmitterConfig for the 1st parameter of io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56) at io.druid.emitter.kafka.KafkaEmitterModule.getEmitter(KafkaEmitterModule.java:56) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.emitter.kafka.KafkaEmitterModule) while locating com.metamx.emitter.core.Emitter annotated with @com.google.inject.name.Named(value=kafka) at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:118) at io.druid.server.emitter.EmitterModule.configure(EmitterModule.java:78) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.server.emitter.EmitterModule)

1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) at com.google.inject.Guice.createInjector(Guice.java:99) at com.google.inject.Guice.createInjector(Guice.java:73) at com.google.inject.Guice.createInjector(Guice.java:62) at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:401) at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:62) at io.druid.cli.ServerRunnable.run(ServerRunnable.java:49) at io.druid.cli.Main.main(Main.java:108) Caused by: java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.emitter.kafka.KafkaEmitterConfig] value failed: bootstrap.servers can not be null at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) at io.druid.guice.JsonConfigurator.configurate(JsonConfigurator.java:103) at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:200) at io.druid.guice.JsonConfigProvider.get(JsonConfigProvider.java:80) at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) at io.druid.guice.SupplierProvider.get(SupplierProvider.java:52) at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61) at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:402) at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) at io.druid.server.emitter.EmitterModule$EmitterProvider.findEmitter(EmitterModule.java:142) at io.druid.server.emitter.EmitterModule$EmitterProvider.inject(EmitterModule.java:120) at io.druid.server.emitter.EmitterModule$EmitterProvider$$FastClassByGuice$$622b285e.invoke() at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:54) at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:89) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:132) at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:93) at com.google.inject.internal.MembersInjectorImpl$1.call(MembersInjectorImpl.java:80) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:80) at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:223) at com.google.inject.internal.Initializer.injectAll(Initializer.java:132) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:174) ... 8 more

jon-wei commented 6 years ago

I did a bit of investigation, all of the configuration properties are null when trying to instantiate KafkaEmitterConfig. I haven't figured out a resolution, but I believe the problem is related to the following:

jon-wei commented 6 years ago

Just saw that a PR was opened for this but closed since it broke other things: https://github.com/druid-io/druid/pull/5201

quenlang commented 6 years ago

@jon-wei Does it will merge into Druid 0.12.0 ?

nishantmonu51 commented 6 years ago

@jon-wei have fixed the failing test and reopened the PR. please review.

nishantmonu51 commented 6 years ago

@quenlang moved to milestone 0.12.0

jon-wei commented 6 years ago

Fixed by #5201