search

druid-io / tranquility

Tranquility helps you send real-time event streams to Druid and handles partitioning, replication, service discovery, and schema rollover, seamlessly and without downtime.
Apache License 2.0
514 stars 229 forks source link

cannot add extension druid-datasketches ? #309

Open longhopefor opened 4 years ago

longhopefor commented 4 years ago

tranquility 0.8.2 druid 0.16.0

run: bin/tranquility server -configFile conf/server.json -Ddruid.extensions.loadList='["druid-datasketches"]' -Ddruid.extensions.directory=/usr/local/druid-0.16.0/extensions

in server.json, i add:

"metricsSpec": [ { "name" : "count", "type" : "count" }, { "name":"sketch_uid", "type":"thetaSketch", "fieldName":"uid", "isInputThetaSketch":"false", "size":"1048576" } ]

output: 2019-10-14 09:19:46,634 [main] INFO i.d.initialization.Initialization - Loading extension [druid-datasketches] for class [io.druid.initialization.DruidModule] 2019-10-14 09:19:46,701 [main] INFO i.d.initialization.Initialization - added URL[file:/usr/local/druid-0.16.0/extensions/druid-datasketches/commons-math3-3.6.1.jar] 2019-10-14 09:19:46,701 [main] INFO i.d.initialization.Initialization - added URL[file:/usr/local/druid-0.16.0/extensions/druid-datasketches/druid-datasketches-0.16.0-incubating.jar] 2019-10-14 09:19:46,701 [main] INFO i.d.initialization.Initialization - added URL[file:/usr/local/druid-0.16.0/extensions/druid-datasketches/memory-0.12.2.jar] 2019-10-14 09:19:46,702 [main] INFO i.d.initialization.Initialization - added URL[file:/usr/local/druid-0.16.0/extensions/druid-datasketches/sketches-core-0.13.4.jar] 2019-10-14 09:19:46,702 [main] INFO i.d.initialization.Initialization - added URL[file:/usr/local/druid-0.16.0/extensions/druid-datasketches/slf4j-api-1.7.25.jar] 2019-10-14 09:19:47,070 [main] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true] 2019-10-14 09:19:47,545 [main] WARN io.druid.segment.indexing.DataSchema - No metricsSpec has been specified. Are you sure this is what you want? 2019-10-14 09:19:47,547 [main] WARN io.druid.segment.indexing.DataSchema - No metricsSpec has been specified. Are you sure this is what you want? 2019-10-14 09:19:47,550 [main] INFO c.metamx.emitter.core.LoggingEmitter - Start: started [true] 2019-10-14 09:19:47,552 [main] WARN io.druid.segment.indexing.DataSchema - No metricsSpec has been specified. Are you sure this is what you want? java.lang.IllegalArgumentException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory] at [Source: N/A; line: -1, column: -1] (through reference chain: Object[][1]) at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) at com.metamx.tranquility.druid.DruidBeams$.makeFireDepartment(DruidBeams.scala:433) at com.metamx.tranquility.druid.DruidBeams$.fromConfigInternal(DruidBeams.scala:299) at com.metamx.tranquility.druid.DruidBeams$.fromConfig(DruidBeams.scala:204) at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:118) at com.metamx.tranquility.server.http.ServerMain$$anonfun$2.apply(ServerMain.scala:98) at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143) at com.metamx.common.scala.collection.package$MapLikeOps$$anonfun$strictMapValues$1.apply(package.scala:143) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) at scala.collection.immutable.Map$Map3.foreach(Map.scala:161) at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at com.metamx.common.scala.collection.package$MapLikeOps.strictMapValues(package.scala:143) at com.metamx.tranquility.server.http.ServerMain$.createServlet(ServerMain.scala:98) at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:168) at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167) at scala.Option.foreach(Option.scala:257) at com.twitter.app.App$class.nonExitingMain(App.scala:167) at com.metamx.tranquility.server.http.ServerMain$.nonExitingMain(ServerMain.scala:49) at com.twitter.app.App$class.main(App.scala:133) at com.metamx.tranquility.server.http.ServerMain$.main(ServerMain.scala:49) at com.metamx.tranquility.distribution.DistributionMain$.main(DistributionMain.scala:34) at com.metamx.tranquility.distribution.DistributionMain.main(DistributionMain.scala) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type id 'thetaSketch' into a subtype of [simple type, class io.druid.query.aggregation.AggregatorFactory] at [Source: N/A; line: -1, column: -1] (through reference chain: Object[][1]) at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) at com.fasterxml.jackson.databind.DeserializationContext.unknownTypeException(DeserializationContext.java:862) "format": "json" at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:167) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:99) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:151) at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:17) at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124) at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769)

partsalliance commented 4 years ago

I am having the same issue with quantilesDoublesSketch.

indrekj commented 4 years ago

Same problem here, @longhopefor @partsalliance were you able to solve it?

From the logs it seems that the extension itself is loaded:

May 28 09:11:48 ip-10-4-0-31 tranquility[24856]: 2020-05-28 09:11:48,904 [main] INFO  io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=tr
ue, directory='/usr/local/druid/extensions', hadoopDependenciesDir='hadoop-dependencies', hadoopContainerDruidClasspath='null', loadList=[druid-datasketches]}]
May 28 09:11:48 ip-10-4-0-31 tranquility[24856]: 2020-05-28 09:11:48,930 [main] INFO  i.d.initialization.Initialization - Loading extension [druid-datasketches] for class [io.druid.initialization.DruidModule]
May 28 09:11:48 ip-10-4-0-31 tranquility[24856]: 2020-05-28 09:11:48,941 [main] INFO  i.d.initialization.Initialization - added URL[file:/usr/local/druid/extensions/druid-datasketches/druid-datasketches-0.18.1.jar]
May 28 09:11:48 ip-10-4-0-31 tranquility[24856]: 2020-05-28 09:11:48,941 [main] INFO  i.d.initialization.Initialization - added URL[file:/usr/local/druid/extensions/druid-datasketches/commons-math3-3.6.1.jar]
longhopefor commented 4 years ago

i don't use it now, i guess the tranquility is not compatible with the latest druid