A Proposal about Extension Center

[FrankChen021]

Motivation

Druid provides lots of useful extensions, but there're some problems here:

1. many of the extensions are provided by community and not are shipped with Druid release.
   That means if we want to use kafka-emitter for example, we have to download that extension on every node in cluster

2. Most of core extensions are not enabled or loaded by default. If we need to enable one, people have to modify the configuration file, add that extension to druid.extensions.loadList and restart the nodes one by one

3. Extensions are not well known, especially community contributed ones. These extensions lay in document which are not checked frequently. One day, I checked the doc about extensions, found that there were so many useful extensions there I hadn't known before, like moving-average-query, kafka-emitter. Had I knew them earlier, I would introduce them in production.

Event at the time I knew that moving-average-query exactly matched my need, I still gave up due to the first two problems above

So, I think maybe we could introduce an Extension Center, something like AppStore, to allow people discover/download/upgrade/enable/disable extensions dynamically.

Proposed changes

I haven't come up with a complete solution how to implement it, I know it's not easy to do it, apparently it involves many parts to do it. And the most import one is the UI, the most complex part is how to keep the configurations related to extensions so that they could be taken effect dynamically, and how to dynamic enable extensions.

Before diving into details, maybe we could discuss if its viable

[suneet-s]

+1 for the overall idea of making it easier to manage and deploy extensions - both core and custom.

I think having an extension center for any extension within Druid can be problematic because we would need to deal with cases where extensions can only be enabled after the service is restarted. I don't think Druid should be in charge of re-starting itself. Whatever orchestration tool is being used by Druid should decide what strategy is best to bring the new extension online - eg. rolling restart, hard restart, shadow deployment, etc. Because of this, even if an Extension Center exists, Druid operators will need another component to manage their Druid cluster.

I also think that we will need to reconcile the configuration on the machine with the configuration that a user provided in the extension center. I don't know if this should become part Druid's core competencies.

I still think there might be value in an Extension Center, I just haven't thought about it for long enough to outline the pros and cons and decide what needs to be part of Druid, and what functionality Druid should pull in from other systems. If you have a user workflow in mind that might help with the discussion.

[FrankChen021]

To introduce Extension Center into druid, I think the first problem needed to solved is to centralize configurations. Extensions don't depend on any configuration files distributed in every node. When user installs an extension, user have to provide required configs and these configs will be saved in the centralized configuration center. Every node read configurations from the center. This solves the 2nd problem mentioned by @suneet-s .

There're many such solutions to do it. For example, we can use zookeeper to keep configurations and every node listens changes on configuration node to take configuration effect immediately. This is also how Kafka implements dynamic configuration on topics.

As the 1st problem, I think maybe we can divide extensions into two categories, one requires re-start, the other doesn't. Many extensions in my opinion could be installed without a restart, such as emitter extensions, query extensions. For those required a restart, I agree that the restart should left to user. But before restart, Extension Center has done most of arduous work(download/config) to enable an extension.

[paul-rogers]

@FrankChen021, we can look to see how other Apache tools accomplish the same goals. At the very lowest level, Presto/Trino uses the Java Service Loader as a standard way to load extensions, perhaps in their own class loader to minimize conflicts.

Extensions need access to configuration, as described above. Provide this via a simple Druid-defined API to hide the implementation details. (Seems we currently use Guice to wire up configuration info, but this would be awkward for extensions.)

Another question about configuration is whether it is static (loaded from config files on process launch), or dynamic (able to change in a running system.) Netflix has a mechanism to handle dynamic configuration. There are probably others.

How are extensions configured? Do they need their own config, or can the config be part of the overall Druid config? Drill uses the latter: the config system is robust enough to allow third-party config to live in either the main Druid config files, in an extension-provided config file (for defaults), or in the user customized config file.

Yet another question is how to distribute extensions. Must they be part of the Druid install so that, say, one has to build a new Docker container for each? Or, can they be dynamically distributed the way that Spark pushes job jars to worker nodes? If part of the container, can we configure them so that they are outside of the Druid product directory so one can easily replace an extension, or Druid, without having to rebuild the combined Druid + extensions combination? Drill, for example, provides a "user config" directory that holds config and extension jars so one can upgrade Drill by deleting the old version and installing a new one without mixing in user files.

The title "Extension Center" sounds like a web service: an app store. This suggests that extensions can be added to (or removed from, or upgraded on) a running Druid cluster. They would be installed, then configured, then enabled. To upgrade, we'd disable them, upgrade them, perhaps adjust the config, and enable again.

Maybe start with cleaning up the basic mechanisms, then, when that tidied up, think about the dynamic aspects.