[FEATURE]Getting Started

TackAdam commented 3 months ago

Is your feature request related to a problem? One of the biggest hurdles customers face on-boarding to OpenSearch is understanding the process for the ingestion of their data.

What solution would you like?

Getting Started should provide a tutorial for data ingestion to the customer. The user should be able to select their type of data and be given a step by step process for how to link their data to OpenSearch. It should be easily assessable and clear upon entering OpenSearch Dashboards. UI Design: getting-started drawio

Architecture: getting-started-architecture drawio

Proposed Schema: ` /// Add the new schema proposed based on the existing tutorials schema

`

Do you have any additional context? Catalog: https://github.com/opensearch-project/opensearch-catalog/issues/165 https://github.com/opensearch-project/opensearch-catalog/pull/166

Example wireframe: https://www.figma.com/design/FY1AMyCa7MIKw8klf1u7W0/Trineo-2024?node-id=2819-127551&t=p7csFHIEiUp2iXJo-0

YANG-DB commented 3 months ago

Additional comments:

Each Getting started catalog component is directly associated with an integration component
Getting started tutorial will have an accompanied (if possible) docker compose containing:
- O/S an ingestion agent
- A service that generates the telemetry
- Synthetic loader generator (https://locust.io/)
Each Getting started catalog component will have a few ingestion flavours :
- Fluent-Bit based
- Data-prepper based
- Otel collector based
Using the existing catalog schema and repository for adding the getting started tutorial
Allowing to unify many parts of the existing integration framework is desired

YANG-DB commented 3 months ago

@TackAdam please review suggested schema here lets continue discussion in schema in the catalog issue itself

YANG-DB commented 2 months ago

Getting Started Landing Page

Getting Started Tutorial Selection Page

YANG-DB commented 2 months ago

Each integration will contain a getting-started section that is associated with the context of that service / artiufact.


{
  "name": "otel-services",
  "version": "1.0.0",
  "displayName": "Otel Services Demo Flow",
  "description": "Otel Services Demo Use Case with OpenSearch Observability",
  "license": "Apache-2.0",
  "type": "traces",
  "labels": [
    "log",
    "traces",
    "metrics"
  ],
  "author": "OpenSearch",
  "sourceUrl": "https://github.com/opensearch-project/dashboards-observability/tree/main/server/adaptors/integrations/__data__/repository/otel-services/info",
 ...

  "getting-started": {
    "ingestion": ["otel-collector","data-prepper"],
    "structured": "true",
    "technology": "OpenTelemetry",
    "protocol": ["otel"],
    "live-sample": "true",
    "workflows": [
      {
        "name": "Otel-Collector",
        "description": "This is a Otel-Collector based getting started instructions tutorial",
        "steps": [
          {
            "name": "Fluent-Bit Parser",
            "type": "console-cmd",
            "phase": "ingestion",
            "label": "Log Parsing",
            "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/nginx/getting-started/fluent-bit/parsers.conf"],
            "description": "Setup Fluent-Bit parser config file parsing Nginx access log fields",
            "content": "..."
          },
          {
            "name": "Fluent-Bit Log Converter",
            "type": "console-cmd",
            "phase": "transformation",
            "label": "Log Parsing",
            "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/nginx/getting-started/fluent-bit/otel-converter.lua"],
            "description": "Setup Fluent-Bit logs converter lua script config file converting Nginx access log into Simple schema format",
            "content" ..."
          },
          {
            "name": "Fluent-Bit Setup",
            "type": "console-cmd",
            "phase": "ingestion",
            "label": "Agent Set-Up",
            "info": ["https://github.com/opensearch-project/opensearch-catalog/tree/main/integrations/observability/nginx/getting-started/fluent-bit/fluent-bit.conf"],
            "description": "Setup Fluent-Bit conf file including logs parsing and OpenSearch access",
            "input-params": [
              {
                "name": "opensearch-node1",
                "description": "Opensearch host",
                "type": "host-name"
              },
              {
                "name": "ss4o_logs-nginx-prod",
                "description": "logs sink index name",
                "type": "index-name"
              }
            ],
            "content": "[SERVICE]\n    ...."
          }
        ]
      },
     ....

The getting-started has the following metadata features that can be filtered and grouped by:

"ingestion": list of ingestion options for this asset (example fluent-bit)
"structured": is this telemetry signal structured (bool)
"technology": a list of related technologies associated with this asset
"protocol": a list of protocols the signal comforms with (in case this is structured)
"live-sample": has this getting-stared accompanied with a live (docker) sample

Other metadata fields will be taken from the integration metadata content itself

The general getting started suggested schema https://github.com/opensearch-project/opensearch-catalog/issues/165

TackAdam commented 2 months ago

For the 2.16 release, only enabled by the nav change feature flag. Visible within the Observability work group. Overview landing page: