[8.10] Undefined `log.file.*` fields breaking tests for `filestream` inputs

[ebeahan]

Summary

A recent feature in Beats for 8.10 adds file information to each event. Integrations using filestream inputs are failing their system testing against the 8.10 snapshot if the integration doesn't define the fields.

Example

Results of running elastic-package test system -v for trendmicro/deep_security against 8.10-SNAPSHOT:

--- Test results for package: trendmicro - START ---
FAILURE DETAILS:
trendmicro/deep_security filestream:
[0] field "log.file.device_id" is undefined
[1] field "log.file.inode" is undefined

╭────────────┬───────────────┬───────────┬────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────╮
│ PACKAGE    │ DATA STREAM   │ TEST TYPE │ TEST NAME  │ RESULT                                                                                             │  TIME ELAPSED │
├────────────┼───────────────┼───────────┼────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ trendmicro │ deep_security │ system    │ filestream │ FAIL: one or more errors found in documents stored in logs-trendmicro.deep_security-ep data stream │ 2m12.0360715s │
╰────────────┴───────────────┴───────────┴────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────╯
--- Test results for package: trendmicro - END   ---
Done
Error: one or more test cases failed

Depends on https://github.com/elastic/beats/issues/36695

Impacted integrations

Appears to impact several integration with system tests using type: filestream (list may be incomplete).

obs-infraobs-integrations: https://github.com/elastic/integrations/pull/7716

• [x] apache_tomcat
• [x] coredns
• [x] oracle: https://github.com/elastic/integrations/pull/8087

security-external-integrations: https://github.com/elastic/integrations/pull/8014

• [x] cisco_ise
• [x] cisco_nexus
• [x] f5_bigip
• [x] fortinet_fortimail
• [x] fortinet_fortimanager
• [x] hid_bravura_monitor
• ~[ ] juniper_junos (deprecated)~
• ~[ ] juniper_netscreen (deprecated)~
• [x] juniper_srx
• [x] keycloak
• [x] mysql_enterprise
• [x] sysmon_linux
• [x] trendmicro

obs-cloud-monitoring : https://github.com/elastic/integrations/pull/8068

• [x] docker
• [x] istio
• [x] kubernetes
• [x] nginx_ingress_controller

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

There are more integrations using filestream. They are:

• [x] docker
• [x] istio
• [x] kubernetes
• [x] nginx_ingress_controller

[bhapas]

Depends on https://github.com/elastic/beats/issues/36695

[ebeahan]

Also observing log.file.vol, log.file.idxlo, and log.file.idxhi showing as unmapped fields in the elastic_agent.filebeat for a Windows 2022 host running Elastic Agent.

{
...
    "service.name": "filebeat",
    "log": {
      "file": {
        "path": "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-b0c688\\logs\\elastic-agent-20230928.ndjson",
        "vol": 373091089,
        "idxlo": 1753,
        "idxhi": 4390912
      },
      "offset": 9310202,
      "source": "winlog-default"
    },
...
}

Should these be grouped into this issue? Or I can file a separate bug.

[bhapas]

Also observing log.file.vol, log.file.idxlo, and log.file.idxhi showing as unmapped fields in the elastic_agent.filebeat for a Windows 2022 host running Elastic Agent.

{
...
    "service.name": "filebeat",
    "log": {
      "file": {
        "path": "C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-b0c688\\logs\\elastic-agent-20230928.ndjson",
        "vol": 373091089,
        "idxlo": 1753,
        "idxhi": 4390912
      },
      "offset": 9310202,
      "source": "winlog-default"
    },
...
}

Should these be grouped into this issue? Or I can file a separate bug.

@ebeahan May be a bug on filebeat in beats repo with reference to this issue

[bhapas]

@ebeahan @andrewkroh Should we update the packages that are deprecated too? juniper_junos (deprecated) juniper_netscreen (deprecated)

[andrewkroh]

Do they have tests that are failing? If not, then my opinion is to leave the deprecated packages as is. Worst case is that for 8.10 those fields are mapped as numbers instead of keywords.

[ebeahan]

I agree with @andrewkroh about leaving the deprecated packages alone.

[bhapas]

Fixed with #7716 #8014 #8068 #8087

[andrewkroh]

More #9582