search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.41k stars 3.69k forks source link

BUG - Please check the MySQL connector version 5.1.48 is in the classpath - Druid 0.21.0 #11315

Closed Thelin90 closed 3 years ago

Thelin90 commented 3 years ago

Hello.

I am currently working on Apache Druid on K8S.

I want to be able to do some lookups towards a MYSQL we host in RDS.

Some more context to the problem:

Currently in druid the classpath is set to (And I can verify this inside the pod):

/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/*:

However since version 0.20.02: https://newreleases.io/project/github/apache/druid/release/druid-0.20.2

They also require you to set some parameters for the jdbc.

They also mention here:

That you need the jar to be put in:

I can see in the logs, the jar is picked up via the logs:

2021-05-28T10:42:36,636 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.21.0.jar
2021-05-28T10:42:36,651 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.21.0.jar
2021-05-28T10:42:36,653 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: mapdb-1.0.8.jar, postgresql-42.2.14.jar, druid-lookups-cached-global-0.21.0.jar, mysql-connector-java-5.1.48.jar
2021-05-28T10:42:36,746 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-0.21.0.jar, postgresql-42.2.14.jar
2021-05-28T10:42:36,748 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-s3-extensions], jars: druid-s3-extensions-0.21.0.jar
2021-05-28T10:42:36,753 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.7.jar, zstd-jni-1.3.3-1.jar, kafka-clients-2.7.0.jar, druid-kafka-indexing-service-0.21.0.jar, lz4-java-1.7.1.jar
2021-05-28T10:42:36,757 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.21.0.jar
2021-05-28T10:42:36,760 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-single], jars: antlr-2.7.7.jar, mapdb-1.0.8.jar, druid-lookups-cached-single-0.21.0.jar, postgresql-42.2.14.jar, stringtemplate-3.2.jar, mysql-connector-java-5.1.48.jar
2021-05-28T10:42:37,035 INFO [main] org.apache.druid.cli.CliCoordinator - Coordinator is configured to act as Overlord as well (druid.coordinator.asOverlord.enabled = true).
2021-05-28T10:42:37,563 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.21.0.jar
2021-05-28T10:42:37,575 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.21.0.jar
2021-05-28T10:42:37,596 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: mapdb-1.0.8.jar, postgresql-42.2.14.jar, druid-lookups-cached-global-0.21.0.jar, mysql-connector-java-5.1.48.jar
2021-05-28T10:42:37,602 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-0.21.0.jar, postgresql-42.2.14.jar
2021-05-28T10:42:37,607 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-s3-extensions], jars: druid-s3-extensions-0.21.0.jar
2021-05-28T10:42:37,615 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.7.jar, zstd-jni-1.3.3-1.jar, kafka-clients-2.7.0.jar, druid-kafka-indexing-service-0.21.0.jar, lz4-java-1.7.1.jar
2021-05-28T10:42:37,625 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.21.0.jar
2021-05-28T10:42:37,642 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-single], jars: antlr-2.7.7.jar, mapdb-1.0.8.jar, druid-lookups-cached-single-0.21.0.jar, postgresql-42.2.14.jar, stringtemplate-3.2.jar, mysql-connector-java-5.1.48.jar
2021-05-28T10:42:55,053 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/FastInfoset-1.2.15.jar:lib/log4j-core-2.8.2.jar:lib/druid-processing-0.21.0.jar:lib/httpclient-4.5.10.jar:lib/jetty-server-9.4.39.v20210325.jar:lib/druid-server-0.21.0.jar:lib/maven-model-3.1.1.jar:lib/plexus-interpolation-1.19.jar:lib/jackson-jaxrs-base-2.10.2.jar:lib/netty-3.10.6.Final.jar:lib/netty-codec-http-4.1.48.Final.jar:lib/httpcore-4.4.11.jar:lib/airline-0.7.jar:lib/calcite-linq4j-1.21.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/joda-time-2.10.5.jar:lib/resilience4j-core-1.3.1.jar:lib/jetty-servlets-9.4.39.v20210325.jar:lib/zookeeper-3.4.14.jar:lib/slf4j-api-1.7.25.jar:lib/netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:lib/jersey-servlet-1.19.3.jar:lib/jetty-util-ajax-9.4.39.v20210325.jar:lib/RoaringBitmap-0.9.0.jar:lib/commons-net-3.6.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/spymemcached-2.12.3.jar:lib/netty-handler-4.1.48.Final.jar:lib/curator-framework-4.3.0.jar:lib/jersey-guice-1.19.3.jar:lib/jaxb-runtime-2.3.1.jar:lib/caffeine-2.8.0.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/icu4j-55.1.jar:lib/commons-dbcp2-2.0.1.jar:lib/druid-indexing-service-0.21.0.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/audience-annotations-0.5.0.jar:lib/javax.servlet-api-3.1.0.jar:lib/config-magic-0.9.jar:lib/druid-sql-0.21.0.jar:lib/jsr305-2.0.1.jar:lib/jetty-client-9.4.39.v20210325.jar:lib/jetty-util-9.4.39.v20210325.jar:lib/commons-math3-3.6.1.jar:lib/maven-settings-3.1.1.jar:lib/jvm-attach-api-1.5.jar:lib/avatica-core-1.17.0.jar:lib/netty-codec-4.1.48.Final.jar:lib/derbynet-10.14.2.0.jar:lib/jersey-core-1.19.3.jar:lib/vavr-0.10.2.jar:lib/j2objc-annotations-1.1.jar:lib/jcodings-1.0.43.jar:lib/google-api-client-1.26.0.jar:lib/commons-compress-1.19.jar:lib/maven-settings-builder-3.1.1.jar:lib/antlr4-runtime-4.5.1.jar:lib/netty-common-4.1.48.Final.jar:lib/google-http-client-jackson2-1.26.0.jar:lib/log4j-jul-2.8.2.jar:lib/aether-impl-0.9.0.M2.jar:lib/checker-qual-2.5.7.jar:lib/druid-indexing-hadoop-0.21.0.jar:lib/derby-10.14.2.0.jar:lib/commons-io-2.6.jar:lib/commons-logging-1.1.1.jar:lib/curator-recipes-4.3.0.jar:lib/jna-4.5.1.jar:lib/jersey-server-1.19.3.jar:lib/maven-aether-provider-3.1.1.jar:lib/datasketches-memory-1.2.0-incubating.jar:lib/netty-handler-proxy-4.1.48.Final.jar:lib/vavr-match-0.10.2.jar:lib/google-oauth-client-1.26.0.jar:lib/error_prone_annotations-2.3.2.jar:lib/netty-codec-dns-4.1.48.Final.jar:lib/json-path-2.3.0.jar:lib/guice-multibindings-4.1.0.jar:lib/commons-text-1.3.jar:lib/rhino-1.7.11.jar:lib/log4j-1.2-api-2.8.2.jar:lib/druid-aws-common-0.21.0.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-jaxb-annotations-2.10.2.jar:lib/commons-lang3-3.8.1.jar:lib/commons-compiler-3.0.11.jar:lib/datasketches-java-1.3.0-incubating.jar:lib/jetty-http-9.4.39.v20210325.jar:lib/netty-buffer-4.1.48.Final.jar:lib/maven-model-builder-3.1.1.jar:lib/aws-java-sdk-kms-1.11.199.jar:lib/txw2-2.3.1.jar:lib/maven-repository-metadata-3.1.1.jar:lib/calcite-core-1.21.0.jar:lib/netty-transport-native-unix-common-4.1.48.Final.jar:lib/zstd-jni-1.3.3-1.jar:lib/curator-x-discovery-4.3.0.jar:lib/jakarta.activation-api-1.2.1.jar:lib/commons-lang-2.6.jar:lib/asm-7.1.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/avatica-metrics-1.17.0.jar:lib/druid-hll-0.21.0.jar:lib/commons-collections4-4.2.jar:lib/derbyclient-10.14.2.0.jar:lib/google-http-client-1.26.0.jar:lib/log4j-api-2.8.2.jar:lib/hibernate-validator-5.2.5.Final.jar:lib/druid-gcp-common-0.21.0.jar:lib/joni-2.1.27.jar:lib/jackson-dataformat-smile-2.10.2.jar:lib/aws-java-sdk-core-1.11.199.jar:lib/guava-16.0.1.jar:lib/plexus-utils-3.0.24.jar:lib/netty-resolver-4.1.48.Final.jar:lib/javax.el-3.0.0.jar:lib/jmespath-java-1.11.199.jar:lib/jaxb-api-2.3.1.jar:lib/xz-1.8.jar:lib/tesla-aether-0.0.5.jar:lib/validation-api-1.1.0.Final.jar:lib/avatica-server-1.17.0.jar:lib/jackson-annotations-2.10.2.jar:lib/jackson-module-guice-2.10.2.jar:lib/aether-spi-0.9.0.M2.jar:lib/shims-0.9.0.jar:lib/disruptor-3.3.6.jar:lib/fastutil-8.2.3.jar:lib/cron-scheduler-0.1.jar:lib/commons-collections-3.2.2.jar:lib/accessors-smart-1.2.jar:lib/jdbi-2.63.1.jar:lib/druid-console-0.21.0.jar:lib/jackson-core-2.10.2.jar:lib/commons-beanutils-1.9.4.jar:lib/javax.activation-1.2.0.jar:lib/druid-core-0.21.0.jar:lib/sigar-1.6.5.132.jar:lib/aether-api-0.9.0.M2.jar:lib/netty-reactive-streams-2.0.0.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/netty-transport-4.1.48.Final.jar:lib/javax.el-api-3.0.0.jar:lib/async-http-client-2.5.3.jar:lib/curator-client-4.3.0.jar:lib/compress-lzf-1.0.4.jar:lib/extendedset-0.21.0.jar:lib/javax.activation-api-1.2.0.jar:lib/guice-servlet-4.1.0.jar:lib/jetty-io-9.4.39.v20210325.jar:lib/ion-java-1.0.2.jar:lib/guice-assistedinject-4.1.0.jar:lib/asm-analysis-7.1.jar:lib/jackson-dataformat-cbor-2.10.2.jar:lib/netty-resolver-dns-4.1.48.Final.jar:lib/jetty-rewrite-9.4.39.v20210325.jar:lib/commons-codec-1.13.jar:lib/janino-3.0.11.jar:lib/classmate-1.1.0.jar:lib/commons-pool2-2.2.jar:lib/netty-codec-socks-4.1.48.Final.jar:lib/esri-geometry-api-2.2.0.jar:lib/lz4-java-1.7.1.jar:lib/protobuf-java-3.11.0.jar:lib/log4j-slf4j-impl-2.8.2.jar:lib/jetty-servlet-9.4.39.v20210325.jar:lib/json-smart-2.3.jar:lib/aws-java-sdk-ec2-1.11.199.jar:lib/jboss-logging-3.2.1.Final.jar:lib/jackson-jaxrs-smile-provider-2.10.2.jar:lib/reactive-streams-1.0.2.jar:lib/javax.inject-1.jar:lib/aether-util-0.9.0.M2.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/jetty-proxy-9.4.39.v20210325.jar:lib/jline-0.9.94.jar:lib/jackson-datatype-guava-2.10.2.jar:lib/jackson-jaxrs-json-provider-2.10.2.jar:lib/aws-java-sdk-s3-1.11.199.jar:lib/wagon-provider-api-2.4.jar:lib/opencsv-4.6.jar:lib/jetty-security-9.4.39.v20210325.jar:lib/jsr311-api-1.1.1.jar:lib/jetty-continuation-9.4.39.v20210325.jar:lib/maven-artifact-3.6.0.jar:lib/druid-services-0.21.0.jar:lib/metrics-core-4.0.0.jar:lib/asm-tree-7.1.jar:lib/jackson-datatype-joda-2.10.2.jar:lib/jackson-databind-2.10.5.1.jar:lib/asm-commons-7.1.jar:lib/stax-ex-1.8.jar:lib/aggdesigner-algorithm-6.0.jar:lib/guice-4.1.0.jar:lib/aopalliance-1.0.jar:lib/okhttp-1.0.2.jar:lib/mysql-connector-java-5.1.48.jar:

However, we still get the error:

��error�Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: Failed to find MySQL driver class. Please check the MySQL connector version 5.1.48 is in the classpath
 at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: -1, column: 354] (through reference chain: org.apache.druid.query.lookup.LookupsState["toLoad"]->java.util.LinkedHashMap["TEST_TOURNAMENT_LOOKUP_LOCAL_MYSQL"]->org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]->org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])��]}
org.apache.druid.java.util.common.IOE: Bad update request to [http://10.1.11.240:8083/druid/listen/v1/lookups/updates] : [400] : [Bad Request]  Response: [:)
��error�Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: Failed to find MySQL driver class. Please check the MySQL connector version 5.1.48 is in the classpath
 at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: -1, column: 354] (through reference chain: org.apache.druid.query.lookup.LookupsState["toLoad"]->java.util.LinkedHashMap["TEST_TOURNAMENT_LOOKUP_LOCAL_MYSQL"]->org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]->org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])��]
    at org.apache.druid.server.lookup.cache.LookupCoordinatorManager$LookupsCommunicator.updateNode(LookupCoordinatorManager.java:834) ~[druid-server-0.21.0.jar:0.21.0]
    at org.apache.druid.server.lookup.cache.LookupCoordinatorManager.doLookupManagementOnNode(LookupCoordinatorManager.java:663) ~[druid-server-0.21.0.jar:0.21.0]
    at org.apache.druid.server.lookup.cache.LookupCoordinatorManager.lambda$lookupManagementLoop$2(LookupCoordinatorManager.java:590) ~[druid-server-0.21.0.jar:0.21.0]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]

All these logs above come from the coordinator pod, I can see the MYSQL jar available is indeed available under:

lib/
extensions/druid-lookups-cached-global/
ls -lah extensions/druid-lookups-cached-single/

Which confirms the params:

  # Extension
  druid.extensions.loadList = '["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-s3-extensions", "druid-kafka-indexing-service", "druid-basic-security", "druid-lookups-cached-global", "druid-lookups-cached-single"]'

  # Lookups
  druid.access.jdbc.enforceAllowedProperties = true
  druid.access.jdbc_allowedProperties = '["useSSL", "requireSSL", "ssl", "sslmode"]'
  druid.access.jdbc_allowUnknownJdbcUrlFormat = true
  druid.lookup.lookupTierIsDatasource = false
  druid.lookup.lookupTier = __default

Based on the link: https://druid.apache.org/docs/latest/configuration/index.html

I have also been able to create a fully local k8s deployment, with a MYSQL docker running outside of the cluster, to replicate the cloud environment, where I can debug this in realtime.

I went more steps further and investigated the source code:

NonRegisteringDriver driver = new NonRegisteringDriver();

Where is seems this fails, and throws the error, I made sure that NonRegisteringDriver do exist in the JAR:

jar tf mysql-connector-java-5.1.48.jar
com/mysql/jdbc/NonRegisteringDriver.class

I would be very happy, if somebody can explain to me why this does not work.

What is missing here?

Thelin90 commented 3 years ago

Update

Problem solved by setting the extensions to the classpath by overriding the druid.sh.

After then running this, it was clear that not only the coordinator, but all components needed this JAR, which seems a bit strange but however, problem solved.

Docs does not say which components that needs this JAR.