Support automatic plugin loading in Data Prepper core

[dlvenable]

Is your feature request related to a problem? Please describe.

Data Prepper plugins can depend on other Data Prepper plugins. They load these plugins manually using the PluginModel.

This results in duplicated logic such as the following:

https://github.com/opensearch-project/data-prepper/blob/91b6666512805bf502186bd683ae800b0943ba10/data-prepper-plugins/aggregate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/aggregate/AggregateProcessor.java#L83-L87

More importantly, it is also not compatible with our new support for schema and documentation generation.

Describe the solution you'd like

Provide a feature in Data Prepper core for loading plugins on behalf of other plugins. Data Prepper can have a new @UsesDataPrepperPlugin annotation which states that the type is a Data Prepper plugin.

In plugin configurations, use the desired plugin interface type. For example, replace

https://github.com/opensearch-project/data-prepper/blob/91b6666512805bf502186bd683ae800b0943ba10/data-prepper-plugins/aggregate-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/aggregate/AggregateProcessorConfig.java#L31-L34

with:

    @JsonProperty("action")
    @NotNull
    @UsesDataPrepperPlugin
    private AggregateAction aggregateAction;

When loading plugins in Data Prepper core, detect this annotation and load the plugin.

[chenqi0805]

@dlvenable I have a slightly different proposal which makes implementation easier without intrusive change in data-prepper-core:

We will keep the original

@JsonProperty("action")
@NotNull
@UsesDataPrepperPlugin(pluginType = AggregateAction.class)
private PluginModel aggregateAction; 

Then we will register a custom deserializer and schema generator for PluginModel.class.

• The schema generator will identify the PluginModel type, retrieves its pluginType in the annotation and then pull the schemas of each option (implementation on AggregateAction) model configuration.
• The custom deserializer will validate PluginModel::name against the allowed pluginName in the pluginType of the annotation (allowed plugin names can be retrieved through reflection since they are implementations of the AggregateAction interface)