Add `pipeline:` field to Elastic Serverless Forwarder specification

[renzedj]

Use Case

As the administrator of our Elastic Cloud environment, I need the capability for my developers to specify a custom Elasticsearch Ingest pipeline for their logs in the ESF config.yml, similar to Elastic Agent or Functionbeat.

Describe the Enhancement

Currently, in order to use a with Elastic Serverless Forwarder with a custom pipeline, it is necessary to deploy a custom log template (e.g., logs-foo.log@custom) This is not required with Elastic Agent or Functionbeat, which allow this to be done with the pipeline: specification. Please add this capability to the Elastic Serverless Forwarder.

An example of what this might look like is:

---
inputs:
  - type: 'cloudwatch-logs'
    id: '<arn>'
    json_content_type: 'ndjson'
    pipeline: 'my-pipeline' # <-- custom Elasticsearch ingest pipeline
    outputs:
      - type: 'elasticsearch'
        args:
          elasticsearch_url:  https://elastic-cloud:443 
          api_key: 'my-api-key'
          es_datastream_name: 'logs-myapp.log-default'
          batch_max_actions: 500
          batch_max_bytes: 10485760

[tommyers-elastic]

hey @renzedj - please could you expand a little on this request?

are we talking here about Elasticsearch ingest pipelines here? is the equivalent behaviour in Elastic Agent you mention this configuration in functionbeat?

if so, it's possible to configure this behaviour without this setting in the lambda, using index templates (option 1 here).

thanks

[renzedj]

Sorry if I wasn't clear. I knew exactly what I was thinking. 🤣

Yes - I'm talking about Elasticsearch ingest pipelines. And yes - the link you provide to the functionbeat documentation is the equivalent functionality to what I'm requesting.

As I noted in my original post, I'm aware that this can be done using index templates (and this is how I'm currently doing it). However, when using a custom data stream, adding a custom template adds another step into the deployment process and requires that I either give my developers admin permissions that I'd rather not give them or that our Elastic admins are engaged. We're trying to enable our developers to manage their own log provisioning as much as possible without granting them overly-broad permissions.

According to the documentation, it appears that all that is needed to send a document through my-pipeline while being indexed is to add ?pipeline=my-pipeline to the URL. I suspect this is how Elastic Agent and Functionbeat do it, so I would assume that it should work similarly for ESF?

We use this functionality for our custom logs with Elastic Agent and it works well.

Thx.

[tommyers-elastic]

thanks for the details - really helpful. adding this to the backlog as an ER.

[jacoor]

When can we expect that ? I also need this functionality

[renzedj]

bump