[FEATURE] Integration loading workflow

Is your feature request related to a problem?

Load Integration

As part of the Integration Ux workflow, once the Integration plugin has loaded and was selected by the user for loading into the system - the B/E should initiate the loading process and display the appropriate status to reflect the loading steps...

This phase follows the previous step in which the user has filtered the Integrations from the repository and selected a specific one to load into the system

Integration loading state machine

 - LOADING
   - VALIDATION
     - UPLOAD
       - READY

The API needed from the backend should be as follows:

Store API:

POST _integration/store/$instance_name

The $instance_name represents the specific name the integration was instanciated with - for example an Nginx Integration can be a template for multiple Nginx instances each representing different domain / aspect such as geographic.

For example the next observability integration:

  ...
  "collection":[
    {
      "logs": [{
        "info": "access logs",
        "input_type":"logfile",  
        "dataset":"nginx.access", <<- subject to user changes
        "namespace": "prod",<<- subject to user changes
        "labels" :["nginx","access"],
        "schema": "file:///.../schema/logs/access.json"
      },
    ...

During the UX interaction with the user, user can update data-stream details shown here: Screenshot 2023-03-01 at 11 28 00 AM

If the user keeps all the original data-stream naming convention (namespace and domain) the next phase would be the validation of the integration prior to loading all the assets.

Data-Stream naming update

In case the user wants to update the data-stream naming details - the next screens will be presented:

Screenshot 2023-03-01 at 11 28 39 AM

Selection of the naming convention may also display available existing data-streams that are selectable if the user wants to choose from available ones.

Validate Integration

Once the Integration instance was stored in the integration store index, it will have a loading status as displayed in the first image.

Next the integration instance will undergo a validation phase in which

assets will be validated with the schema to match fields to the mapping
assets that depend on index-pattern will expect the index to exist
datasource will be validated to verify connection is accessible
mapping templates are verified to exist

If any of the validation failed - the API (the call to _integration/store/$instance_name ) will return a status indicating the current state of the integration:

Response:

{
  "instance": "nginx-prod",
  "integration-name": "nginx",
  "status": "maintenance",
  "issues": [
    { 
      "asset": "dashboard",
      "name": "nginx-prod-core",
      "url": "file:///.../nginx/integration/assets/nginx-prod-core.ndjson",
      "issue": [
        "field cloud.version is not present in mapping sso_log-nginx-prod"
      ]
    }
  ]
}

The next screen shows the maintenance issues:

Screenshot 2023-03-06 at 11 47 06 AM

Once all the issues are manually resolved by the User, the UX can continue the loading process by the next API PUT _integration/store/$instance_name/activate

This API attempts to move the state of the integration to Ready and returns the result status of this call.

Response:

{
  "instance": "nginx-prod",
  "integration-name": "nginx",
  "status": "loading"
}

Load Assets

The loading assets phase will use the existing bulk load api for all the existing assets of the integration

Visualizations
Dashboards
SaveQueries
MappingTemplates

The User can cherry pick specific assets to load and use the next UX window for this purpose

Using the next API PUT _integration/store/$instance_name/load

{
  "instance": "nginx-prod",
  "integration-name": "nginx",
  "assets" :{
    "dashboards": ["nginx-prod-core.ndjson"],
    "savedQueries": ["AvgQueriesPerDayFilter.json"]
  }
}

This is the integration status screen showing some load-failure for assets Screenshot 2023-03-06 at 11 47 06 AM

The user can continue research the failures and attempt fixing them:

Screenshot 2023-03-06 at 11 47 18 AM

Once he continues by using retry button, the appropriate API would be called _integration/store/$instance_name/activate and the loading process would continue.

Once all steps are completed, the status of the integration would become Ready and the Ux can pull this info using status API :

GET _integration/store/$instance_name/status

Response:

{
  "instance": "nginx-prod",
  "integration-name": "nginx",
  "status": "ready",
 "assets":[
       "nginx-prod-core.ndjson":"https://127.0.0.1:5601/app/dashboards#/view/7adfa750-4c81-11e8-b3d7-01146121b73d"
       ...
   ]
}

The ready response returns all the links to the saved objects stored as part of the upload phase under assets field.

Integration Load workflow

Screenshot 2023-03-01 at 7 17 31 PM

What solution would you like? B/E & F/E support for the next API & user flow.

What alternatives have you considered? N/A

Do you have any additional context?

The UX full happy flow for loading an integration nginx-integration-ui-mock

opensearch-project / observability