custom logs integration onboarding guide

[mukeshelastic]

How to onboard logs for which we don't have elastic agent based integration

The custom logs integration documentation doesn't really describe all the steps user needs to take to onboard custom logs in elastic stack. Given the importance of custom log on-boarding use case for Elastic's users and community, we should create custom logs on-boarding guide and include steps such as: how to choose the right type of custom integration ( log, http api, tcp/udp etc, what to configure for each type, how to parse logs into fields, how to set up any other required plumbing such templates, settings, mapping etc.

[amitkanfer]

Related slack conversation: https://elastic.slack.com/archives/C01QQ449KE1/p1666749376816439

[lucabelluccini]

It is a very poor UX.

When the user specifies a data set name like the following example:

The data will end up in the data stream logs-somedatasetname-default (default might change if the namespace is different).

But the integration seems to be designed to install the index template logs-log.log, which matches the index patterns logs-log.log-*.

The UI even proposes to use custom mappings and ingest pipelines, but this will be applied to the Index template logs-log.log, BUT it will not match logs-somedatasetname-default. The logs (default Index Template will be used) See:

See GET _data_stream/logs-somedatasetname-default will return:

{
  "data_streams": [
    {
      "name": "logs-somedatasetname-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-logs-somedatasetname-default-2023.01.06-000001",
          "index_uuid": "L8ZNssqbSJCKHhIuDANtlA"
        }
      ],
      "generation": 1,
      "_meta": {
        "description": "default logs template installed by x-pack",
        "managed": true
      },
      "status": "GREEN",
      "template": "logs", <<<<<<<<
      "ilm_policy": "logs",
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    }
  ]
}

Even if we do not customize the dataset name, the user will end up having data in logs-generic-default which is not obvious nor written anywhere and it will not match the template logs-logs.logs.

If we cannot automate those steps, I would propose to auto-generate the Index Template request body based on the data which has been input and give the option to the user to install it or copy the whole request.

[lucabelluccini]

It seems starting custom logs 2.0.0 integration, at least we generate the correct assets https://github.com/elastic/integrations/pull/5347