dbwiddis
commented
1 year ago The Node.js client would continue to be the low-level client that it is and will continue to expose all OpenSearch endpoints. In addition, a mechanism would be built to expose the non-core capabilities of the OpenSearch that are made available via plugins and extensions.
I really like this approach, not just for Dashboards, but as a standard that Extensions should adhere to for compatibility with languages beyond Java / js.
dblock
joshuarrrr
AMoo-Miki
wbeckler
Haitham-Shalaby
OpenSearch Dashboards: Decoupling the rigid dependency
Background
By definition, decoupling refers to running activities separate from each other or allowing them to not develop in the same way (Cambridge Business English Dictionary). The OpenSearch Dashboards: Decoupling initiative wishes to target both of those definitions.
Situation
When OpenSearch Dashboards is run, it compares its version with those of the configured OpenSearch instance and all plugins.
### *OpenSearch Dashboards: Version matching* When OpenSearch Dashboards’ CLI is triggered via [src/cli/dist.js](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/cli/dist.js#L33) with no sub-commands * Control is passed to [src/cli/cli.js](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/cli/cli.js#L49) where [Commander](https://www.npmjs.com/package/commander) is [asked](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/cli/cli.js#L71) to run the serve sub-command. * The serve sub-command is defined in [src/cli/serve/serve.js](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/cli/serve/serve.js#L148) and its action is instructed to [call](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/cli/serve/serve.js#L226) bootstrap. * The bootstrap method, defined in [src/core/server/bootstrap.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1dda7309c2f9dcfdac2d1f52b121b55aed991bfa/src/core/server/bootstrap.ts#L59), then [sets up and starts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1dda7309c2f9dcfdac2d1f52b121b55aed991bfa/src/core/server/bootstrap.ts#L130) the top-level entry point for the server. * The setup method of [src/core/server/root/index.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/root/index.ts#L71) calls [setup](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/root/index.ts#L64) on [an instance](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/root/index.ts#L56) of Server. #### *With Engine* * The setup method of [src/core/server/server.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L123) calls [setup](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L159) on [an instance](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L109) of OpenSearchService. * The setup method of [src/core/server/opensearch/opensearch_service.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/e6bbb405441c7d59a8d5df6a477c5e432fa88b87/src/core/server/opensearch/opensearch_service.ts#L85) makes a [call](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/e6bbb405441c7d59a8d5df6a477c5e432fa88b87/src/core/server/opensearch/opensearch_service.ts#L94) pollOpenSearchNodesVersion. * pollOpenSearchNodesVersion of [src/core/server/opensearch/version_check/ensure_opensearch_version.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/349652685f84aa0c058315d846b110fcf8d23ee2/src/core/server/opensearch/version_check/ensure_opensearch_version.ts#L237) gets the version of the configured OpenSearch engine and passes it along with the version of OpenSearch Dashboards to [mapNodesVersionCompatibility](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/349652685f84aa0c058315d846b110fcf8d23ee2/src/core/server/opensearch/version_check/ensure_opensearch_version.ts#L161): so they can be compared by [opensearchVersionCompatibleWithOpenSearchDashboards](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/349652685f84aa0c058315d846b110fcf8d23ee2/src/core/server/opensearch/version_check/ensure_opensearch_version.ts#L183). * opensearchVersionCompatibleWithOpenSearchDashboards of src/core/server/opensearch/version_check/opensearch_opensearch_dashboards_version_compatability.ts considers any OpenSearch Engine and OpenSearch Dashboards incompatible: * if they are not within the same major version, or * when the major versions match, if Engine has a lower minor version than Dashboard. * The result is passed back to the setup of OpenSearchService to [calculateStatus$](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/e6bbb405441c7d59a8d5df6a477c5e432fa88b87/src/core/server/opensearch/opensearch_service.ts#L115) and set the status level to [ServiceStatusLevels.critical](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/opensearch/status.ts#L60). #### *With Plugins* * The setup method of [src/core/server/server.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L123) calls [discover](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L130) on [an instance](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/server.ts#L107) of PluginsService. * The discover method of [src/core/server/plugins/plugins_service.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1dda7309c2f9dcfdac2d1f52b121b55aed991bfa/src/core/server/plugins/plugins_service.ts#L110) makes a [call](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/1dda7309c2f9dcfdac2d1f52b121b55aed991bfa/src/core/server/plugins/plugins_service.ts#L115) to an imported discover method. * The discover method of [src/core/server/plugins/discovery/plugins_discovery.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugins_discovery.ts#L63) locates plugins and [calls](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugins_discovery.ts#L83) createPlugin$ for each one. * [createPlugin$](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugins_discovery.ts#L198) then [calls](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugins_discovery.ts#L204) parseManifest on the plugin’s manifest file. * A plugin manifest file is always named [opensearch_dashboards.json](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L48) at the root of a plugin and contains information such as its id, version, and ... . * parseManifest of [src/core/server/plugins/discovery/plugin_manifest_parser.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L86) reads in opensearchDashboardsVersion , or in its absence version, from the manifest file and [compares](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L167) it to the version of OpenSearch Dashboards using isVersionCompatible. * [isVersionCompatible](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L233) normalizes the versions to drop any possible suffixes and evaluates them as incompatible: * if the versions are not identical * The result is passed back to the parseManifest of [src/core/server/plugins/discovery/plugin_manifest_parser.ts](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L167) to [throw an exception](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/src/core/server/plugins/discovery/plugin_manifest_parser.ts#L168) if needed.opensearchDashboardsVersionand it is not identical to the version of Dashboards, an exception is thrown and the process exits.opensearchDashboardsVersionand the plugin’s version it is not identical to the version of Dashboards, an exception is thrown and the process exits.Problems
The self-imposed rules are based on the idea of minimizing the compatibility matrix to reduce the amount of surface to test and maintain. This not taking advantage of semantic versioning has complicated development and releases, and is potentially an inhibitor to rapid adoption of Dashboards and plugins.
This compatibility check logic does not match the promises we have made by adopting semantic versioning:
As OpenSearch, releasing a new version of OpenSearch and OpenSearch Dashboards requires updating the manifests for all plugins and consequently bumping their versions, even if they have no other changes. In fact, the release automation blindly overwrites the
opensearchDashboardsVersionvalue in the manifest file of the plugins (ref).For a contributor, it is an annoyance to have to edit the manifests of plugins to make them employable for testing during development of OpenSearch Dashboards and other plugins. If a contributor rebases the upstream main branch of Dashboards on to their development branch and it includes a patch version bump, they are forced to update all the plugins before bootstrapping. This is substantially more irritating for someone maintaining a staging instance since they need to individually build all the plugins and install them again, even those that had no code changes.
An administrator of an OpenSearch instance might put off upgrading their instances of OpenSearch due their delicate data but OpenSearch Dashboards, being just a UI, could be updated without an impact on their OpenSearch; while on OpenSearch 2.2.0, one might want to allow their users to enjoy the new Wizard functionality which was stabilized in Dashboards 2.3.0. This potential upgrade path is blocked by the current compatibility checks.
OpenSearch, OpenSearch Dashboards, and plugin repositories bump their versions as soon as their development cycles start after a release. Since we follow the train model, if a repository is not ready for a release, a previous release is pulled into the release pipeline and its version is bumped. Even though a tag is cut, the git history is contaminated with conflicting histories where the same version is used for completely different codebases. While the conflicting histories isn’t a big detractor of our reputation, this process is an unwelcome complication for our release process.
Decoupling OpenSearch Dashboards and OpenSearch
Various components of OpenSearch Dashboards interact with OpenSearch using one of three unrelated methods: the legacy client, the OpenSearch client, and direct HTTP calls.
* elasticsearch@16.7.3 ([npm](https://www.npmjs.com/package/@elastic/elasticsearch)) * via core/server/opensearch/legacy * via plugins/data_source/server/legacy * directly by @osd/opensearch-archiver * to provide type definitions in * plugins/data/server/search * plugins/data/public/search * plugins/data/common/search * plugins/data_source/server/client * plugins/discover/public/application * plugins/vis_type_vega/server * plugins/visualizations/server * some tests * @opensearch-project/opensearch ([npm](https://www.npmjs.com/package/@opensearch-project/opensearch)) * via core/server/opensearch/client * via plugins/data_source/server/client * to provide type definitions in * plugins/data/server/search * plugins/data/common/search * OsdClientRequester ([github](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/packages/osd-dev-utils/src/osd_client/osd_client_requester.ts#L89)) making HTTP calls * via @osd/dev-utilsProposed Solution
Make OpenSearch Dashboards rely solely on the Node.js client, @opensearch-project/opensearch, and delegate the responsibility of interacting with OpenSearch to it. The Node.js client is a low-level library that could be compatible with more that one major version of OpenSearch and any variations would be normalized by a wrapper (introduced below). OpenSearch Dashboards will be instructed to use the latest compatible major version of the Node.js client available. This involves replacing the components that consume the legacy elasticsearch client and the direct HTTP channel with method that call the official Node.js client.
Addendum
The legacy elasticsearch@^16.7.0 has been used by all releases of OpenSearch Dashboards so far. Since OpenSearch Dashboards never depended on the previously deprecated and now removed type mapping, its can use @opensearch-project/opensearch@^2 to communicate with any of the existing releases of OpenSearch and since both, the engine and the client, follow semantic versioning, if there is a breaking change that @opensearch-project/opensearch needs to adapt to in the future, it will not cause a breaking change for OpenSearch Dashboards v2. In fact, the Node.js client, as of v2.1.0, only checks if the distribution name is “opensearch” and does not care for the version (ref). As such, theoretically, any version of OpenSearch Dashboards can function against any version of OpenSearch.
Decoupling OpenSearch Dashboards and Plugins
OpenSearch Dashboards plugins directly import public objects exposed by Dashboards’ source code to build and function. While the Extensions SDK promises to eliminate a direct dependency on OpenSearch Dashboards, some of these plugins might choose not to switch to extensions due their need for certain privileges or APIs not exposed by the extensions SDK. Since the majority of the plugins import directly from source files of OpenSearch Dashboards, no promises can be made that they will not be impacted adversely by changes in OpenSearch Dashboards’ code.
Proposed Solution
If OpenSearch Dashboards would want to continue to support external core plugins, a mechanism would be created to expose the type definitions of all public objects of OpenSearch Dashboards that the developers would like to make accessible. During development of plugins, the consumption of any objects not deliberately exposed by OpenSearch Dashboards would be prevented by linting rules. At runtime and during their initialization, plugins will access the type definitions exposed by OpenSearch Dashboards to verify that they are compatible with that version of OpenSearch Dashboards and would silently fail if they are not; this will allow for OpenSearch Dashboards to gracefully degrade.
Decoupling the Node.js Client and OpenSearch
The Node.js client, @opensearch-project/opensearch (npm), is a low-level communication facilitator between applications running with JavaScript and the OpenSearch. The current version (v2.1.0) of the Node.js client does not discriminate based on the version of OpenSearch and as long as the engine advertises itself as “opensearch,” it is deemed compatible; in other words, it is pretty much decoupled already.
Proposed Solution
The Node.js client would continue to be the low-level client that it is and will continue to expose all OpenSearch endpoints. In addition, a mechanism would be built to expose the non-core capabilities of the OpenSearch that are made available via plugins and extensions. As a result, as long as the OpenSearch’s APIs or the client dependencies don’t introduce breaking changes, there would be no need for a major version bump.
Decoupling OpenSearch Dashboards and the Node.js Client
At the time of building or installing an instance of OpenSearch Dashboards, the package manager, Yarn, pulls in the highest version of the Node.js client that satisfies the requirements of OpenSearch Dashboards; this is trusting that the Node.js client is following semantic versioning. OpenSearch Dashboards would expect the basic index creation/removal and document retrieval APIs never change within a major version of the client.
Proposed Solution
With an eye on making OpenSearch Dashboards an independent product, a new wrapper would be created to take on the job of interacting with the low-level OpenSearch Node.js client. The wrapper would be a high-level abstraction layer that would service the needs of OpenSearch Dashboards with normalized responses. The wrapper will recognize the features made available to it by the OpenSearch instance, in order to allow OpenSearch Dashboards to gracefully degrade. The wrapper will also be responsible for adjusting its communication recipes for variations of a feature across versions of OpenSearch.
When OpenSearch Dashboard starts up, it would instruct the wrapper to establish a connection with OpenSearch via the client. The wrapper would make a few requests to ascertain that it is indeed capable of understanding what it gets back from the OpenSearch instance and would then attempt to execute a few more requests to determine the variations in the APIs and feature unavailability.
This design would allow wrappers to be created for engines other than OpenSearch to facilitate visualization and analysis of a wider collection of data sources using OpenSearch Dashboards (image below).
Decoupling OpenSearch Dashboards and the Extension SDKs
The Extensions SDK could possibly have internal mechanisms to normalize calls against older versions of OpenSearch Dashboards APIs known to the SDK. However, this will not enable interactions between an extension and the APIs exposed by a later version of OpenSearch Dashboards which increases the burden on extensions developers to keep up with newer releases of the Extensions SDK.
Proposed Solution
At runtime and during the initialization of an extension, if the Extensions SDK it was built with does not recognize the version of OpenSearch Dashboards, it would access the type definitions exposed by OpenSearch Dashboards to verify that the APIs consumed by the extensions match those the Extensions SDK knows how to handle; an initialization error would be triggered if the Extensions SDK does not know how to communicate with any of the APIs required by the extension. The Extensions SDK might also choose to facilitate optional consumption of APIs by the extension to allow the extension to detect missing functionality and degrade gracefully.
Discussion
Version Detection
A technique frequently employed for managing dependencies is versioning and the same can technically be employed for detecting the compatibility of APIs exposed by OpenSearch Dashboards and data storage/search engines. When breaking changes are introduced in any API, a major version bump would have to be introduced, marking everything as incompatible. Versioning APIs will only help reduce the frequency of version bumps for plugins, SDK, and extensions; it will not help decouple them from OpenSearch Dashboards.
let’s consider an example where OpenSearch Dashboards provides APIs A, B, and C at v1 and Plugin Y uses APIs A and B at v1. If API C is deprecated or if its signature changes enough to make it not backwards-compatible, OpenSearch Dashboards will bump the APIs to v2, making Plugin Y incompatible even though the APIs that it used never changed. If the Extensions SDK relies on versioned APIs, an Extension X, also only consuming APIs A and B, and all other extensions will have to be recompiled with a new version of the SDK to continue to function on the newer OpenSearch Dashboards release. Instead, if the SDK depends on detecting the capabilities of OpenSearch Dashboards and the signature of its APIs, Extension X would continue to work as expected even though it was compiled with an older SDK.
Feature Detection
The main component of decoupling OpenSearch Dashboards and data storage/search engines is the introduction of the wrapper that serves both the targets of decoupling by allowing OpenSearch Dashboards to function independent of a specific data storage/search engine (where each engine will get its own wrapper) and to not be locked into a specific version of an engine, by the wrapper taking on the job of modifying calls and normalizing responses. The key to both of these is for the wrapper to recognize the capabilities and limitations of the specific version of the engine it is communicating with. If the OSD-OS-wrapper relied solely on detecting the version of the engine, older releases of OpenSearch Dashboards will not be compatible with newer releases of OpenSearch simply because the wrapper wouldn’t know how to handle the newer version of OpenSearch. Relying on feature detection, the older OpenSearch Dashboards will continue to function until the wrapper is unable to effectively communicate or understand the engine, or when OpenSearch Dashboards is told that an API it relies on is not normalizable by the current wrapper.
While we could simply enrich the current low-level OpenSearch Node.js client with the OpenSearch Dashboards wrapper, that choice could limit the involvement of the open-source community in creating other OpenSearch Dashboards wrappers using existing clients of other data storage/search engines.
Decoupling Shared Dependencies
OpenSearch Dashboards and plugin/extensions could have common dependencies. For core plugins, during bootstrapping and build, a mechanism is enforced to have all shared dependencies be employed with a common version; this undesirable enforcement is not sustainable with decoupling. For dependencies that are imported and used by Node.js runtime, NPM and Yarn don’t hoist the different versions of shared dependencies and instead place them in the
node_modulesfolder of the component. However, for those dependencies that are bundled by webpack and sent to the browser, since their inclusion is only by name, conflicts are inevitable. One technique to avoid this is to use NPM/Yarn aliases for the dependencies of each extension while building them. This could be further enhanced by merging dependencies that have matching or compatible signatures. The negative impact of being able to enforce common dependencies is that the bundle payload size increases.Design