search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.53k stars 3.71k forks source link

add an integration test with extension diretories as symlinks so that we can validate the version bump for commons-io #11393

Open zachjsh opened 3 years ago

zachjsh commented 3 years ago

An extension loading issue was found when using commons-io version 2.9.0. In particular When the an extension directory is configured as a symlink, the symlink is not walked properly, and no jars are found, resulting in the extension not being loaded properly. An example of such a configuration for the druid-hdfs-storage extension is shown below:

  % pwd
  /Users/user1/apache-druid-0.22.0-SNAPSHOT/extensions
  %
  % ls -al
  total 0
  drwxr-xr-x  29 user1 staff   928 Jun 29 12:38 .
  drwxr-xr-x  12 user1  staff   384 Jun 29 12:35 ..
  drwxr-xr-x  28 user1  staff   896 Jun 29 12:35 druid-avro-extensions
  drwxr-xr-x  13 user1  staff   416 Jun 29 12:35 druid-aws-rds-extensions
  drwxr-xr-x  10 user1  staff   320 Jun 29 12:35 druid-azure-extensions
  drwxr-xr-x   3 user1  staff    96 Jun 29 12:35 druid-basic-security
  drwxr-xr-x   5 user1  staff   160 Jun 29 12:35 druid-bloom-filter
  drwxr-xr-x   4 user1  staff   128 Jun 29 12:35 druid-datasketches
  drwxr-xr-x   3 user1 staff    96 Jun 29 12:35 druid-ec2-extensions
  drwxr-xr-x   4 user1 staff   128 Jun 29 12:35 druid-google-extensions
  lrwxr-xr-x   1 user1 staff    49 Jun 29 12:38 druid-hdfs-storage -> /Users/user1/druid-hdfs-storage
  drwxr-xr-x   3 user1  staff    96 Jun 29 12:35 druid-histogram
  drwxr-xr-x  10 user1  staff   320 Jun 29 12:35 druid-kafka-extraction-namespace
  drwxr-xr-x   7 user1  staff   224 Jun 29 12:35 druid-kafka-indexing-service
  drwxr-xr-x  11 user1  staff   352 Jun 29 12:35 druid-kerberos
  drwxr-xr-x   6 user1  staff   192 Jun 29 12:35 druid-kinesis-indexing-service
  drwxr-xr-x  38 user1  staff  1216 Jun 29 12:35 druid-kubernetes-extensions
  drwxr-xr-x   5 user1  staff   160 Jun 29 12:35 druid-lookups-cached-global
  drwxr-xr-x   7 user1  staff   224 Jun 29 12:35 druid-lookups-cached-single
  drwxr-xr-x  20 user1  staff   640 Jun 29 12:35 druid-orc-extensions
  drwxr-xr-x  15 user1  staff   480 Jun 29 12:35 druid-pac4j
  drwxr-xr-x  24 user1  staff   768 Jun 29 12:35 druid-parquet-extensions
  drwxr-xr-x   3 user1  staff    96 Jun 29 12:35 druid-protobuf-extensions
  drwxr-xr-x  73 user1  staff  2336 Jun 29 12:35 druid-ranger-security
  drwxr-xr-x  13 user1  staff   416 Jun 29 12:35 druid-s3-extensions
  drwxr-xr-x   4 user1  staff   128 Jun 29 12:35 druid-stats
  drwxr-xr-x   3 user1  staff    96 Jun 29 12:35 mysql-metadata-storage
  drwxr-xr-x   4 user1  staff   128 Jun 29 12:35 postgresql-metadata-storage
  drwxr-xr-x   3 user1  staff    96 Jun 29 12:35 simple-client-sslcontext
  %

An issue was filed with the apache commons-io team https://issues.apache.org/jira/browse/IO-741.

We should add an integration test which tests this scenario so that we can bump the version of this dependency in the future with confidence.

jihoonson commented 3 years ago

I removed the starter labels because this is not easy. The blocker here is that our integration tests don't use the same extension loading system, but use a hacky way relying on the maven dependency plugin.