[libbeat] add_nomad_metadata unreliable

Please include configurations and logs if available.

For confirmed bugs, please report:

Version: filebeat 8.14.3
Operating System: Ubuntu 22.04.4 LTS
Steps to Reproduce:
- Configure add_nomad_metadata
- Start a periodic job / batch job in Nomad
- See the logs appear in Elasticsearch, that sometimes do and sometimes do not have the nomad.* fields attached.

Looking at the debug logs of filebeat, it seems like the 'event' from Nomad is sometimes being received/parsed too late by some file watchers. It does sometimes work, just not always and therefore not reliably. We configured Nomad to keep the log files on-disk for a while even after the allocation/process has exited, to make sure Filebeat has proper time to ship the logs - but the experimental add_nomad_metadata seems to have issues attaching said metadata.

- type: filestream
  id: nomad-stdout-logs
  paths:
  - /opt/nomad/alloc_mounts/**/alloc/logs/*.stdout.0
  - /opt/nomad/alloc_mounts/**/alloc/logs/*.stderr.0
  file_identity.path: ~
  parsers:
  - ndjson:
      target: ""
      ignore_decoding_error: true
  processors:
  - dissect:
      tokenizer: "/opt/nomad/alloc_mounts/%{uuid1}-%{uuid2}-%{uuid3}-%{uuid4}-%{uuid5}-%{container.labels.com_hashicorp_nomad_group_name}/alloc/logs/%{log.filename}"
      field: "log.file.path"
      target_prefix: ""
      ignore_fialure: true
      overwrite_keys: false
  - dissect: 
      tokenizer: "%{container.labels.com_hashicorp_nomad_task_name}.%{log.pipe}.0"
      field: "log.filename"
      target_prefix: ""
      ignore_fialure: true
      overwrite_keys: false
  - script:  # Combining the allocation ID from the 5 parts, so I can separate out the group name and allocation ID reliably.
      lang: javascript
      source: >
        function process(event) {
          event.Put("container.labels.com_hashicorp_nomad_alloc_id", event.Get("uuid1") + "-" + event.Get("uuid2") + "-" + event.Get("uuid3") + "-" + event.Get("uuid4") + "-" + event.Get("uuid5"));
        }
  - drop_fields:
      ignore_missing: true
      fields: ["uuid1", "uuid2", "uuid3", "uuid4", "uuid5"]
  - add_nomad_metadata:
      secret_id: "${NOMAD_TOKEN}"
      default_indexers.enabled: true
      default_matchers.enabled: false
      scope: node
      node: "my-node-name"  # The issue of it being unreliable also happened for `scope: cluster`
      matchers:
        - fields:
            lookup_fields: ['container.labels.com_hashicorp_nomad_alloc_id']

All events reliably have the container.labels.com_hashicorp_nomad_alloc_id set to the Nomad allocation ID.

elastic / beats

[libbeat] add_nomad_metadata unreliable #40321