[FEATURE]Getting Started Template catalog

[YANG-DB]

Is your feature request related to a problem?

Today the Integrations catalog are becoming more popular due to its simple installation and opinionated schema governed by the Otel protocol. Each integration has a TryMe button that has a small dataset which functions as a mock data-points for the dashboard to show some minimal visual appearance.

The need for a comprehensive documentation and instruction (including scripts or live docker image) of how to actually add the agent on top of the observed service, how to map the resulting logs into the Otel (simple schema) format and the actual setup of the agent that will be responsible of the data shipping is not covered and very much missing.

What solution would you like? This is a proposal for the content of such getting-started template and format that will lead to a catalog of getting started component that are accompanied with the following :

• Documentation of the agent
• Instruction for setting up the agent
• Documentation service's log format
• Instruction for transforming the log's format into Otel designated format
• Documentation of the agent's shipping method and recommendations
• Instruction for setting up the agent's logs delivery

Each such component will have the following relationships:

• Associated Integrations
• Associated index name / index mapping (related to the signal type)
• Associated Docker-compose (live sample) that showcase all the instructions and can be run and ingest data using some build in mock that activate the monitored service

Do you have any additional context? Here is the suggested getting-started format that can be later processed by a UX visualization and allow customer interaction.

• Metadata details: name, version, description, license ...
• Workflow : list of steps including description and scripts to follow for making the service's log ingested correctly
• statics : list of visual logo to show with this component's UX dialog
• relatedAssets: list of related components that are associated with this service such as integrations, alerts, queries

{
  "name": "nginx",
  "version": "1.0.0",
  "displayName": "Nginx-Logs",
  "description": "Getting Started With Nginx access logs.",
  "license": "Apache-2.0",
  "type": "logs",
  "labels": ["Observability", "Logs", "Fluent-bit"],
  "author": "OpenSearch",
  "sourceUrl": "https://github.com/opensearch-project/dashboards-observability/tree/main/server/adaptors/integrations/__data__/repository/nginx/info",
  "workflow": [
    {
      "name": "Fluent-Bit Parser",
      "label": "Log Parsing",
      "info": "<info URL>",
      "description": "Setup Fluent-Bit parser config file parsing Nginx access log fields",
      "content": "[PARSER]\n    Name   apache\n    Format regex\n    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] \"(?<method>\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\S*)?)?\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \"(?<referer>[^\\\"]*)\" \"(?<agent>[^\\\"]*)\")?$\n    Time_Key time\n    Time_Format %d/%b/%Y:%H:%M:%S %z\n\n[PARSER]\n    Name   nginx\n    Format regex\n    Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] \"(?<method>\\S+)(?: +(?<path>[^\\\"]*?)(?: +\\S*)?)?\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \"(?<referer>[^\\\"]*)\" \"(?<agent>[^\\\"]*)\")\n    Time_Key time\n    Time_Format %d/%b/%Y:%H:%M:%S %z\n"
    },
    {
      "name": "Fluent-Bit Log Converter",
      "label": "Log Parsing",
      "info": "<info URL>",
      "description": "Setup Fluent-Bit logs converter lua script config file converting Nginx access log into Simple schema format",
      "content": "local hexCharset = \"0123456789abcdef\"\nlocal function randHex(length)\n    if length > 0 then\n        local index = math.random(1, #hexCharset)\n        return randHex(length - 1) .. hexCharset:sub(index, index)\n    else\n        return \"\"\n    end\nend\n\nlocal function format_apache(c)\n    return string.format(\n        \"%s - %s [%s] \\\"%s %s HTTP/1.1\\\" %s %s\",\n        c.host,\n        c.user,\n        os.date(\"%d/%b/%Y:%H:%M:%S %z\"),\n        c.method,\n        c.path,\n        c.code,\n        c.size\n    )\nend\n\nlocal function format_nginx(c)\n    return string.format(\n        \"%s %s %s [%s] \\\"%s %s HTTP/1.1\\\" %s %s \\\"%s\\\" \\\"%s\\\"\",\n        c.remote,\n        c.host,\n        c.user,\n        os.date(\"%d/%b/%Y:%H:%M:%S %z\"),\n        c.method,\n        c.path,\n        c.code,\n        c.size,\n        c.referer,\n        c.agent\n    )\nend\n\nlocal formats = {\n    [\"apache.access\"] = format_apache,\n    [\"nginx.access\"] = format_nginx\n}\n\nfunction convert_to_otel(tag, timestamp, record)\n    local data = {\n        traceId=randHex(32),\n        spanId=randHex(16),\n        [\"@timestamp\"]=os.date(\"!%Y-%m-%dT%H:%M:%S.000Z\"),\n        observedTimestamp=os.date(\"!%Y-%m-%dT%H:%M:%S.000Z\"),\n        body=formats[tag](record),\n        attributes={\n            data_stream={\n                dataset=tag,\n                namespace=\"production\",\n                type=\"logs\"\n            }\n        },\n        event={\n            category=\"web\",\n            name=\"access\",\n            domain=tag,\n            kind=\"event\",\n            result=\"success\",\n            type=\"access\"\n        },\n        http={\n            request={\n                method=record.method\n            },\n            response={\n                bytes=tonumber(record.size),\n                status_code=tonumber(record.code)\n            },\n            flavor=\"1.1\",\n            url=record.path\n        },\n        communication={\n            source={\n                address=\"127.0.0.1\",\n                ip=record.remote\n            }\n        }\n    }\n    return 1, timestamp, data\nend\n"
    },
    {
      "name": "Fluent-Bit Setup",
      "label": "Agent Set-Up",
      "info": "<info URL>",
      "description": "Setup Fluent-Bit conf file including logs parsing and OpenSearch access",
      "content": "[SERVICE]\n    Parsers_File parsers.conf\n\n[INPUT]\n    Name forward\n    Port 24224\n\n[FILTER]\n    Name parser\n    Match nginx.access\n    Key_Name log\n    Parser nginx\n\n[FILTER]\n    Name parser\n    Match apache.access\n    Key_Name log\n    Parser apache\n\n[Filter]\n    Name    lua\n    Match   *\n    Script  otel-converter.lua\n    call    convert_to_otel\n\n[OUTPUT]\n    Name  opensearch\n    Match nginx.access\n    Host  opensearch-node1\n    Port  9200\n    Index ss4o_logs-nginx-prod\n    Suppress_Type_Name On\n    tls             On\n    tls.verify      Off\n    HTTP_User       admin\n    HTTP_Passwd     my_%New%_passW0rd!@#\n\n[OUTPUT]\n    Name  opensearch\n    Match apache.access\n    Host  opensearch-node1\n    Port  9200\n    Index ss4o_logs-apache-prod\n    Suppress_Type_Name On\n    tls             On\n    tls.verify      Off\n    HTTP_User       admin\n    HTTP_Passwd     my_%New%_passW0rd!@#\n\n[OUTPUT]\n    Name stdout\n    Match nginx.access"
    }
  ],
  "statics": {
    "logo": [{
      "annotation": "Fluent-Bit Logo",
      "path": "logo.svg"
    },
    {
      "annotation": "Nginx Logo",
      "path": "logo.svg"
    }]
  },
  "relatedAssets": [
    {
      "name": "nginx-integration",
      "type": "integration",
      "version": "1.0.0",
      "displayName": "Nginx-Integration",
      "catalog": "Observability",
      "sourceUrl": "https://github.com/opensearch-project/dashboards-observability/tree/main/server/adaptors/integrations/__data__/repository/nginx/info"
    }
  ],
  "liveDemo": {
    "command": "docker-compose up -d",
    "path": "<docker-compose-url>"
  }
}