OpenSearch Dashboards: Consolidation of clients as part of version decoupling

[manasvinibs]

As part of version decoupling project, one of the pre-req task is to consolidate all the clients usages in Opensearch Dashboards which interacts with Opensearch.

Reference: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3095

Decoupling OpenSearch Dashboards and OpenSearch

Various components of OpenSearch Dashboards interact with OpenSearch using one of three unrelated methods: the legacy client, the OpenSearch client, and direct HTTP calls.

* elasticsearch@16.7.3 ([npm](https://www.npmjs.com/package/@elastic/elasticsearch)) * via core/server/opensearch/legacy * via plugins/data_source/server/legacy * directly by @osd/opensearch-archiver * to provide type definitions in * plugins/data/server/search * plugins/data/public/search * plugins/data/common/search * plugins/data_source/server/client * plugins/discover/public/application * plugins/vis_type_vega/server * plugins/visualizations/server * some tests * @opensearch-project/opensearch ([npm](https://www.npmjs.com/package/@opensearch-project/opensearch)) * via core/server/opensearch/client * via plugins/data_source/server/client * to provide type definitions in * plugins/data/server/search * plugins/data/common/search * OsdClientRequester ([github](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/d7004dc5b0392477fdd54ac66b29d231975a173b/packages/osd-dev-utils/src/osd_client/osd_client_requester.ts#L89)) making HTTP calls * via @osd/dev-utils

Proposed Solution

Make OpenSearch Dashboards rely solely on the Node.js client, @opensearch-project/opensearch, and delegate the responsibility of interacting with OpenSearch to it. The Node.js client is a low-level library that could be compatible with more that one major version of OpenSearch and any variations would be normalized by a wrapper (introduced below). OpenSearch Dashboards will be instructed to use the latest compatible major version of the Node.js client available. This involves replacing the components that consume the legacy elasticsearch client and the direct HTTP channel with method that call the official Node.js client.

Requirement:

• Identify places in the code where all the different client methods are used to connect to Opensearch
  • Legacy client method using elasticsearch-js.
  • Opensearch-js 1.0 client
  • Opensearch-js 2.0 client
  • Http calls
• Refactor all the components with legacy client elasticsearch@16.7.3 npm usages and replace it with opensearch-js client.
• Replace any direct HTTP calls made to Opensearch with official Node.js client.

[manasvinibs]

Following are the list of functions, interfaces and type definitions that are using legacy elasticsearch client properties and methods within the Opensearch Dashboard repo -

1. In src/core/server/opensearch/legacy/api_types.ts -

   1. Contains deprecated interface LegacyCallAPIOptions which defines how Opensearch API call should be made and result be processed.
   2. Contains deprecated interface LegacyAPICaller that defines Opensearch endpoints and makes use of legacy client class methods as API endpoint return types.
   3. Below are the functions where LegacyAPICaller is used to provide type definitions. Instead we should replace it with opensearch-js client caller interface type.
      1. getUsageCollector() - src/plugins/vis_type_vega/server/usage_collector/get_usage_collector.ts
      2. getUsageCollector() - src/plugins/visualizations/server/usage_collector/get_usage_collector.ts
      3. class ValidationTelemetryService - src/plugins/vis_type_timeseries/server/validation_telemetry/validation_telemetry_service.ts
      4. getUsage(), getClusterUuid() - src/plugins/usage_collection/server/routes/stats/stats.ts
      5. bulkFetch(), bulkFetchUsage() - src/plugins/usage_collection/server/collector/collector_set.ts
      6. fetch() - src/plugins/usage_collection/server/collector/collector.ts
      7. StatsCollectionConfig {} interface - src/plugins/telemetry_collection_manager/server/types.ts
      8. getOpenSearchDashboards() - src/plugins/telemetry/server/telemetry_collection/get_opensearch_dashboards.ts
      9. getSavedObjectsCounts() - src/plugins/opensearch_dashboards_usage_collection/server/collectors/opensearch_dashboards/get_saved_object_counts.ts
      10. getQueryClient(), getBasicAuthClient() - src/plugins/data_source/server/legacy/configure_legacy_client.ts
      11. fetchProvider() - src/plugins/data/server/search/collectors/fetch.ts
      12. decideClient() - src/plugins/data/server/index_patterns/routes.ts
      13. resolveTimePattern()- src/plugins/data/server/index_patterns_fetcher/lib/resolve_time_pattern.ts
      14. callIndexAliasApi(), callFieldCapsApi() - src/plugins/data/server/index_patterns/fetcher/lib/opensearch_api.ts
      15. getFieldCapabilities()- src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_capabilities.ts
      16. class IndexPatternsFetcher- src/plugins/data/server/index_patterns/fetcher/index_patterns_fetcher.ts
      17. fetchProvider() - src/plugins/data/server/dql_telemetry/usage_collector/fetch.ts
      18. setupMockCallCluster() - src/plugins/data/server/dql_telemetery/usage_collector/fethc.test.ts
      19. TestOpenSearchServerclass - src/core/test_helpers/osd_server.ts
      20. AllServices class- src/core/server/ui_settings/integration_tests/lib/server.ts
      21. class LegacyScopedClusterClient- src/core/server/opensearch/legacy/scoped_cluster_clients.ts
      22. callAsInternalUser(), callAsCurrentUser() - src/core/server/opensearch/legacy/cluster_client.ts

2. src/core/server/opensearch/legacy/cluster_client.ts -

   1. LegacyClusterClient class creates Elasticsearch client instance to make API calls on behalf of dashboards internal user.
      1. createLegacyClusterClient() - src/core/server/opensearch/opensearch_service.ts 2.ILegacyClusterClient type - Cluster API client created by platform. Below are the interfaces and functions with ILegacyClusterClient type definitions
      2. CollectionConfig{}, Collection{} - src/plugins/telemetery_collection_manager/server/types.ts
      3. registerCollection() - src/plugins/telemetry/server/telemetery_collection/register_collection.ts
      4. OpenSearchServiceSetup{} - src/core/server/opensearch/types.ts
      5. OpenSearchServiceStart{} - src/core/server/opensearch/types.ts
   2. ILegacyCustomClusterClient type - Cluster API client created by plugin. Below are the files with its type definitions -
      1. FetcherTask -src/plugins/telemetry/server/fetcher.ts
      2. OpenSearchServiceStart{} - src/core/server/opensearch/types.ts
      3. OpenSearchServiceSetup{} - src/core/server/opensearch/types.ts
      4. createLegacyCustomClient()- src/core/server/opensearch/opensearch_service.ts
   3. callAsInternalUser()- LegacyAPICaller interface implementation to make Opensearch API call on behalf of Dashboards internal user.
      1. getStatsCollectionConfig() - src/plugins/telemetery_collection_manager/server/plugin.ts
   4. callAsCurrentUser() - LegacyAPICaller interface implementation to make API call on behalf of user initiated request to dashboards server via http request headers.
   5. asScoped() - Elasticsearch client instance shared between all the scoped clients created for the cluster client
      1. CoreOpenSearchRouteHandlerContext class - src/core/server/core_route_handler_context.ts
      2. getStatsCollectionConfig() - src/plugins/telemetery_collection_manager/server/plugin.ts

3. src/plugins/data_source/server/legacy/client_config.ts

   1. parseClientOptions() - Provides config options which is used by legacy Elasticsearch client object to create root client instance. This consumes ConfigOptions module from elasticSearch package which will be replaced by opensearch js client options.

4. src/plugins/data_source/server/legacy/configure_legacy_client.ts - Mainly Data Source plugin consumes this wrapper for legacy client connection to opensearch APIs

   1. configureLegacyClient()- creates root client which is an instance of legacy client class and use it for connection with the given data source endpoint.
      1. DataSourceService class provides legacy data source client as part of the setup() lifecycle methods to the plugins that uses DataSourceService - src/plugins/data_source/server/data_source_service.ts
   2. getBasicAuthClient() - Root client uses legacy elasticsearch client
   3. getAWSClient()
   4. getQueryClient() - Creating root client from legacy elasticsearch for different types of auth types
   5. callAPI() - calls Opensearch API endpoint using legacy client

5. src/plugins/data_source/server/plugins.ts - DataSourcePlugin using DataSourceService to register plugin context to route handler context and create router methods.

   1. context.dataSource.opensearch.legacy.getClient()
      1. registerResolveIndexRoute() - src/plugins/index_pattern_management/server/routes/resolve_index.ts
      2. DataSourcePluginRequestContext{} - src/plugins/data_source/server/types.ts
      3. registerRoutes() - src/plugins/data/server/index_patterns/routes.ts

6. src/core/server/opensearch/opensearch_service.ts - class OpenSearchService which implements CoreService, creates both legacy elasticsearch client as well as new opensearch js client. With legacy client, there are two instances: legacy custom client and legacy client. OpenSearchService setup() lifecycle method takes care of instantiating all the 3 clients before moving on to start() lifecycle. Below are its references:

   1. setup() - src/core/server/server.ts
      1. coreSetup, savedObjectsSetup, statusSetup - Server class initializes all the core services that are part of Opensearch Dashboard and also defines its lifecycle methods.

7. packages/osd-opensearch-archiver/src/opensearch_archiver.ts

   1. createGenerateIndexRecordsStream() - packages/osd-opensearch-archiver/src/lib/indices/generate_index_records_stream.ts
   2. createGenerateDocRecordsStream() - packages/osd-opensearch-archiver/src/lib/docs/generate_doc_records_stream.ts
   3. loadAction() - packages/osd-opensearch-archiver/src/actions/load.ts
      1. createCreateIndexStream() - packages/osd-opensearch-archiver/src/lib/indices/create_index_stream.ts
         1. deleteOpenSearchDashboardsIndices() - packages/osd-opensearch-archiver/src/lib/indices/opensearch_dashboards_index.ts
            1. deleteIndex() →waitForSnapshotCompletion()
            2. fetchOpenSearchDashboardsIndices()
      2. createIndexDocRecordsStream() - packages/osd-opensearch-archiver/src/lib/docs/index_doc_records_stream.ts
      3. createDefaultSpace()
   4. unloadAction()
      1. createDeleteIndexStream() → cleanOpenSearchDashboardsIndices()
   5. emptyOpenSearchDashboardsIndexAction() → migrateOpenSearchDashboardsIndex()

[manasvinibs]

List of Opensearch Dashboards core plugins currently using Elasticsearch legacy client methods:

vis_type_vega
visualizations
vis_type_timeseries
usage_collection
telemetry_collection_manager
opensearch_dashboards_usage_collection
telemetry_collection
data_source
data
index_pattern_management

List of packages in OSD currently using Elasticsearch legacy client methods:

osd-opensearch-archiver

[manasvinibs]

Aliasing elasticsearch to npm:@opensearch-project/opensearch@2.1.0

1. Errors when aliasing "@types/elasticsearch": "npm:@opensearch-project/opensearch@2.1.0" from @osd/opensearch-archiver

yarn osd bootstrap
yarn run v1.22.19
$ scripts/use_node scripts/osd bootstrap
 info [opensearch-dashboards] running yarn

$ scripts/use_node ./preinstall_check
[1/5] Validating package.json...
[2/5] Resolving packages...
warning Resolution field "typescript@4.0.2" is incompatible with requested version "typescript@~4.5.2"
success Already up-to-date.

 succ yarn.lock analysis completed without any issues
 succ 22 bootstrap builds are cached
 info [@osd/opensearch-archiver] running [osd:bootstrap] script
ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 2: /home/ubuntu/.nvm/versions/node/v14.20.1/lib/node_modules/yarn/bin/yarn.js run osd:bootstrap
      error Command failed with exit code 2.
      $ ../../scripts/use_node ../../scripts/remove.js target && tsc
      Deleted files and directories:
         /home/ubuntu/OpenSearch-Dashboards/packages/osd-opensearch-archiver/target
      src/cli.ts(103,9): error TS2345: Argument of type '{ host: string | true | string[]; log: string | never[]; }' is not assignable to parameter of type 'ClientOptions'.
        Object literal may only specify known properties, and 'host' does not exist in type 'ClientOptions'.
      src/lib/docs/generate_doc_records_stream.ts(32,18): error TS2305: Module '"../../../node_modules/@types/elasticsearch"' has no exported member 'SearchParams'.
      src/lib/docs/generate_doc_records_stream.ts(32,32): error TS2305: Module '"../../../node_modules/@types/elasticsearch"' has no exported member 'SearchResponse'.
      src/lib/docs/generate_doc_records_stream.ts(74,15): error TS2769: No overload matches this call.
        Overload 1 of 4, '(params?: Scroll<Record<string, any>> | undefined, options?: TransportRequestOptions | undefined): TransportRequestPromise<ApiResponse<Record<string, any>, unknown>>', gave the following error.
          Argument of type '{ scrollId: any; scroll: string; }' is not assignable to parameter of type 'Scroll<Record<string, any>>'.
            Object literal may only specify known properties, but 'scrollId' does not exist in type 'Scroll<Record<string, any>>'. Did you mean to write 'scroll_id'?
        Overload 2 of 4, '(callback: callbackFn<Record<string, any>, unknown>): TransportRequestCallback', gave the following error.
          Argument of type '{ scrollId: any; scroll: string; }' is not assignable to parameter of type 'callbackFn<Record<string, any>, unknown>'.
            Object literal may only specify known properties, and 'scrollId' does not exist in type 'callbackFn<Record<string, any>, unknown>'.
      src/lib/docs/index_doc_records_stream.ts(61,38): error TS2769: No overload matches this call.
        Overload 1 of 4, '(params?: Bulk<any[]> | undefined, options?: TransportRequestOptions | undefined): TransportRequestPromise<ApiResponse<Record<string, any>, unknown>>', gave the following error.
          Argument of type '{ requestTimeout: number; body: any[]; }' is not assignable to parameter of type 'Bulk<any[]>'.
            Object literal may only specify known properties, and 'requestTimeout' does not exist in type 'Bulk<any[]>'.
        Overload 2 of 4, '(callback: callbackFn<Record<string, any>, unknown>): TransportRequestCallback', gave the following error.
          Argument of type '{ requestTimeout: number; body: any[]; }' is not assignable to parameter of type 'callbackFn<Record<string, any>, unknown>'.
            Object literal may only specify known properties, and 'requestTimeout' does not exist in type 'callbackFn<Record<string, any>, unknown>'.
      src/lib/docs/index_doc_records_stream.ts(62,14): error TS2339: Property 'errors' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/delete_index.ts(49,29): error TS2769: No overload matches this call.
        Overload 1 of 4, '(params?: IndicesGetAlias | undefined, options?: TransportRequestOptions | undefined): TransportRequestPromise<ApiResponse<Record<string, any>, unknown>>', gave the following error.
          Argument of type '{ name: string; ignore: number[]; }' is not assignable to parameter of type 'IndicesGetAlias'.
            Object literal may only specify known properties, and 'ignore' does not exist in type 'IndicesGetAlias'.
        Overload 2 of 4, '(callback: callbackFn<Record<string, any>, unknown>): TransportRequestCallback', gave the following error.
          Argument of type '{ name: string; ignore: number[]; }' is not assignable to parameter of type 'callbackFn<Record<string, any>, unknown>'.
            Object literal may only specify known properties, and 'name' does not exist in type 'callbackFn<Record<string, any>, unknown>'.
      src/lib/indices/delete_index.ts(50,22): error TS2339: Property 'status' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/delete_index.ts(95,7): error TS2339: Property 'snapshots' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/delete_index.ts(106,13): error TS2339: Property 'snapshots' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/generate_index_records_stream.ts(41,29): error TS2769: No overload matches this call.
        Overload 1 of 4, '(params?: IndicesGet | undefined, options?: TransportRequestOptions | undefined): TransportRequestPromise<ApiResponse<Record<string, any>, unknown>>', gave the following error.
          Argument of type '{ index: any; filterPath: string[]; }' is not assignable to parameter of type 'IndicesGet'.
            Object literal may only specify known properties, but 'filterPath' does not exist in type 'IndicesGet'. Did you mean to write 'filter_path'?
        Overload 2 of 4, '(callback: callbackFn<Record<string, any>, unknown>): TransportRequestCallback', gave the following error.
          Argument of type '{ index: any; filterPath: string[]; }' is not assignable to parameter of type 'callbackFn<Record<string, any>, unknown>'.
            Object literal may only specify known properties, and 'index' does not exist in type 'callbackFn<Record<string, any>, unknown>'.
      src/lib/indices/generate_index_records_stream.ts(60,14): error TS2537: Type 'ApiResponse<Record<string, any>, unknown>' has no matching index signature for type 'string'.
      src/lib/indices/opensearch_dashboards_index.ts(31,18): error TS2305: Module '"../../../node_modules/@types/elasticsearch"' has no exported member 'CreateDocumentParams'.
      src/lib/indices/opensearch_dashboards_index.ts(105,6): error TS2339: Property 'map' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/opensearch_dashboards_index.ts(129,24): error TS2769: No overload matches this call.
        Overload 1 of 4, '(params?: DeleteByQuery<{ query: { bool: { must_not: { ids: { type: string; values: string[]; }; }; }; }; }> | undefined, options?: TransportRequestOptions | undefined): TransportRequestPromise<...>', gave the following error.
          Argument of type '{ index: string; body: { query: { bool: { must_not: { ids: { type: string; values: string[]; }; }; }; }; }; ignore: number[]; }' is not assignable to parameter of type 'DeleteByQuery<{ query: { bool: { must_not: { ids: { type: string; values: string[]; }; }; }; }; }>'.
            Object literal may only specify known properties, and 'ignore' does not exist in type 'DeleteByQuery<{ query: { bool: { must_not: { ids: { type: string; values: string[]; }; }; }; }; }>'.
        Overload 2 of 4, '(callback: callbackFn<Record<string, any>, unknown>): TransportRequestCallback', gave the following error.
          Argument of type '{ index: string; body: { query: { bool: { must_not: { ids: { type: string; values: string[]; }; }; }; }; }; ignore: number[]; }' is not assignable to parameter of type 'callbackFn<Record<string, any>, unknown>'.
            Object literal may only specify known properties, and 'index' does not exist in type 'callbackFn<Record<string, any>, unknown>'.
      src/lib/indices/opensearch_dashboards_index.ts(146,14): error TS2339: Property 'total' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/opensearch_dashboards_index.ts(146,29): error TS2339: Property 'deleted' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/opensearch_dashboards_index.ts(149,14): error TS2339: Property 'deleted' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      src/lib/indices/opensearch_dashboards_index.ts(150,14): error TS2339: Property 'total' does not exist on type 'ApiResponse<Record<string, any>, unknown>'.
      info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
          at makeError (/home/ubuntu/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:25045:11)
          at handlePromise (/home/ubuntu/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:23981:26)
          at processTicksAndRejections (internal/process/task_queues.js:95:5)
          at async /home/ubuntu/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:9238:9
          at async scheduleItem (/home/ubuntu/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:11285:9)
error Command failed with exit code 1.

Params type mismatch in Elasticseach client modules vs Opensearch client modules.

1. Properties types missing in new opensearch-js client that exist in legacy elasticsearch:

       Client['deleteTemplate']
       Client['fieldStats']
       Client['getTemplate']
       Client['putTemplate']
       Client['suggest']
       Client['indices']['flushSynced']
       Client['indices']['existsType']

[Haitham-Shalaby]

Hi, Can we consider this issue as done since issue #4041 and #4142 are closed and merged, what is the dependency on #4306

[manasvinibs]

Hi, Can we consider this issue as done since issue #4041 and #4142 are closed and merged, what is the dependency on #4306 Here is the link to the list of places where client update needs to happen - https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3900#issuecomment-1544753815 The PR's above linked are only addressing consolidating clients in osd-opensearch-archiver package.