[Meta] Extensions Framework

[saratvemulapalli]

Add the framework support for OpenSearch SDK to interface with extension points.

Issues

Steps to deliver extensions framework [P0]

• [x] Discover Extensions: #1972
• [x] Add communication framework support for OpenSearch SDK #2019
• [x] Proof of Concept: Add support for first extension point #2711
• [x] Discover Extensions: Read extensions configuration during OpenSearch bootstrap https://github.com/opensearch-project/OpenSearch/issues/3084
• [ ] OpenSearch SDK: #1619
• [x] Initial Benchmark Performance #2231 #3012
• [x] Rest Handlers Extension point: https://github.com/opensearch-project/OpenSearch/issues/3130
• [ ] #3136
• [x] https://github.com/opensearch-project/OpenSearch/issues/3434
• [x] https://github.com/opensearch-project/OpenSearch/issues/3408
• [x] Run AD plugin as an Extension to create a detector - https://github.com/opensearch-project/OpenSearch/issues/3635

Back Burner[P1]

• [ ] Extend extensions framework support for additional extension points to migrate a plugin from opensearch-project #3011

Things we broke [P1]

• [ ] https://github.com/opensearch-project/OpenSearch/issues/2772
• [x] #3282

Good to have [P2]

• [ ] #2228
• [ ] https://github.com/opensearch-project/OpenSearch/issues/3118

Meta: https://github.com/opensearch-project/OpenSearch/issues/1422

[owaiskazi19]

Dependencies on PluginsService for repository-s3 plugin to run independently

1. pluginsService.updatedSettings()
2. pluginsService.filterPlugins(Plugin.class)
3. pluginsService.getPluginSettingsFilter()
4. p.getNamedXContent().stream() 

[saratvemulapalli]

Started working on PluginDiscovery:

1. Create new PluginOrchestrator (which will eventually replace PluginsService.java) to read through new plugins ("pluginsv2", didnt find a better name we could rename it later).
2. Update PluginInfo to understand a new field called Type (which will support LEGACY, INDEPENEDENT, REMOTE (in future)).
3. Define a new manifest file to expose a TCP port and PluginInfo.

[saratvemulapalli]

Feature branch on fork: https://github.com/saratvemulapalli/OpenSearch/tree/sarat-plugin-sandboxing Seeing problems while reading new plugins folder:

[2022-01-12T22:13:52,984][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [3c22fb88f11c.ant.amazon.com] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
    at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
    at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
    at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
    at sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525) ~[?:?]
    at java.nio.file.Files.exists(Files.java:2514) ~[?:?]
    at org.opensearch.plugins.PluginsService.findPluginDirs(PluginsService.java:365) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.plugins.PluginsOrchestrator.pluginsDiscovery(PluginsOrchestrator.java:67) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.plugins.PluginsOrchestrator.<init>(PluginsOrchestrator.java:52) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.node.Node.<init>(Node.java:409) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.node.Node.<init>(Node.java:329) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
    ... 6 more
uncaught exception in thread [main]
java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
    at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
    at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
    at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
    at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
    at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
    at java.base/java.nio.file.Files.exists(Files.java:2514)
    at org.opensearch.plugins.PluginsService.findPluginDirs(PluginsService.java:365)
    at org.opensearch.plugins.PluginsOrchestrator.pluginsDiscovery(PluginsOrchestrator.java:67)
    at org.opensearch.plugins.PluginsOrchestrator.<init>(PluginsOrchestrator.java:52)
    at org.opensearch.node.Node.<init>(Node.java:409)
    at org.opensearch.node.Node.<init>(Node.java:329)
    at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
    at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
    at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
    at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
    at org.opensearch.cli.Command.main(Command.java:101)
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
    at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/logs/opensearch.log