Tutorial Custom pipeline-mapping with integrations

Vince-dS commented 7 months ago

Hi Team, In 8.8 ( same in 8.13 ?), I am testing custom pipeline+mapping along an integration. I am also looking at this document: https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html.

To me, there are couple of points to correct:

Where is the “Select Apply now and rollover.” ?
Once updating a custom pipeline from the integration (advanced options), I do not see such option. I do an API request to trigger the rollover.
```
POST    logs-nginx.access-default/_rollover 
```
the proposition to rollover just after the custom pipeline should add some reminder about the mapping. Why not emphasising to check the custom mapping according to the custom pipeline ? To ensure, both have been tested and are ok before triggering the roll over.

FYI, I was initially suggesting this link to a customer who has moved from on prem. to the cloud with a strong interest to add pipeline/mapping along integrations. Thanks

PS: it is my first doc issue - pls let me know if I need to bring more detail/context.

Vince-dS commented 7 months ago

@kilfoyle , fyi, while working on a training doc on this topic, I noted a reference which is mentioning rollover api instead of “Select Apply now and rollover.” :

https://www.elastic.co/guide/en/observability/current/apm-ilm-how-to.html#apm-data-streams-custom-three

As well, a screenshot from v8.8 :

kilfoyle commented 6 months ago

Hi @Vince-dS I haven't really worked much with data streams or pipelines so I'm learning as I go. I tried to run through this tutorial and make the changes you suggested.

In "Step 2: Apply your ingest pipeline", as you suggest, I can add instructions to test the custom mapping:

Screenshot 2024-04-24 at 12 04 45 PM

Then, I can also add instructions to run a rollover (using the API, since I also don't see any "Apply now and rollover" button).

Screenshot 2024-04-24 at 12 08 52 PM

However, Now, when I run this query:

GET metrics-system.cpu-default/_search
{
  "query": {
    "exists": {
      "field": "test"
    }
  }
}

I get a false result. :-(
I tried waiting 30 minutes and still no luck. If you know what I could be doing wrong I'd appreciate any tips.

{
  "took": 17,
  "timed_out": false,
  "_shards": {
    "total": 4,
    "successful": 4,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 234,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-metrics-system.cpu-default-2024.04.24-000001",
        "_id": "IlCpvz-gHJNfjmqlAAABjxDfbwc",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-04-24T16:10:29.511Z",
          "agent": {
            "ephemeral_id": "e2f05e40-c57c-49cc-b20c-2cc30e2aa2e1",
            "id": "21f5ea72-f5a6-4759-87a3-413d58fc0848",
            "name": "104fb9e65f7d",
            "type": "metricbeat",
            "version": "8.13.2"
          },
          "data_stream": {
            "dataset": "system.cpu",
            "namespace": "default",
            "type": "metrics"
          },

-- cut ---

elastic / ingest-docs

Tutorial Custom pipeline-mapping with integrations #1027