libbeat does not support dynamic: 'runtime' fields.

[NiklasBeierl]

I want to leverage elastic common schema and let filebeat create my index template but add a few fields. If this is not the intended use-case, or there is a different - non-tedious - way to "extend" ECS, please let me know.

When I try to launch filebeat with these settings:

setup.ilm.enabled: false

setup.template:
  name: container-logs
  pattern: container-logs*
  enabled: true
  overwrite: true
  settings.index:
    number_of_replicas: 0
    lifecycle.name: "container-logs"
  append_fields:
    - name: my.field
      type: object
      dynamic: "runtime"

filebeat aborts with:

Attaching to filebeat-container-1
filebeat-container-1  | {"log.level":"info","@timestamp":"2024-01-19T22:13:43.382Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":811},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
filebeat-container-1  | {"log.level":"info","@timestamp":"2024-01-19T22:13:43.389Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":819},"message":"Beat ID: 9e88f991-bc1a-4bcb-9489-81a3c67bf6cb","service.name":"filebeat","ecs.version":"1.6.0"}
filebeat-container-1  | {"log.level":"error","@timestamp":"2024-01-19T22:13:43.390Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1312},"message":"Exiting: unpacking template config fails: 'runtime' is invalid dynamic setting accessing 'setup.template.append_fields.0.dynamic' (source:'filebeat.yml')","service.name":"filebeat","ecs.version":"1.6.0"}
filebeat-container-1  | Exiting: unpacking template config fails: 'runtime' is invalid dynamic setting accessing 'setup.template.append_fields.0.dynamic' (source:'filebeat.yml')
filebeat-container-1 exited with code 0

Interestingly, setting true, false or 'strict' works and creates an appropriate index-template. It is only 'dynamic' that fails.

• Version: 8.12.0
• Operating System: Docker on Arch Linux x86-64

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[NiklasBeierl]

I have tracked this back the definition of the field type

It seems that libbeat is unaware of the "runtime" field type of elasticsearch? :open_mouth:

Is that a wontfix, a technical debt or is the syntax different between elastic and libbeat?