search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
187 stars 392 forks source link

Invalid ECS field usages at root-level #7808

Closed andrewkroh closed 9 months ago

andrewkroh commented 10 months ago

Across packages owned by elastic/security-external-integrations the following fields are being used at the document root, but according to ECS they are only allowed be nested under other ECS namespaces like host or source. These usages need to be changed to align with ECS. And fixing these issues will be required to move to package-spec 3.0.0.

Field Name Usage count
as.number 2
geo.city_name 6
geo.continent_name 4
interface.id 2
interface.name 3
os.family 3
os.name 7
os.type 2
vlan.id 2
x509.issuer.common_name 2

This was detected by looking at fields.yml mappings only. It's possible that the fields are not actually used in some cases. If I accidentally included a deprecated or rsa2elk package then please ignore that field.

Source Locations

Package Data Stream Field Github Source
1password item_usages os.name View Source
1password signin_attempts os.name View Source
azure_frontdoor access geo.city_name View Source
azure_frontdoor access geo.continent_name View Source
azure_frontdoor waf geo.city_name View Source
azure_frontdoor waf geo.continent_name View Source
bluecoat director geo.city_name View Source
carbonblack_edr log os.type View Source
cisco_aironet log interface.id View Source
cisco_meraki events geo.city_name View Source
cisco_meraki log geo.city_name View Source
cloudflare_logpush network_session vlan.id View Source
crowdstrike fdr os.type View Source
f5 bigipafm geo.city_name View Source
f5 bigipapm geo.city_name View Source
fireeye nx interface.name View Source
infoblox_nios log interface.name View Source
juniper_junos log geo.city_name View Source
juniper_netscreen log geo.city_name View Source
juniper_srx log as.number View Source
juniper_srx log geo.city_name View Source
juniper_srx log geo.continent_name View Source
juniper_srx log interface.id View Source
juniper_srx log interface.name View Source
juniper_srx log os.family View Source
juniper_srx log os.name View Source
juniper_srx log vlan.id View Source
juniper_srx log x509.issuer.common_name View Source
lyve_cloud audit os.name View Source
netflow log as.number View Source
netflow log geo.city_name View Source
netflow log geo.continent_name View Source
netflow log os.family View Source
netflow log os.name View Source
panw panos x509.issuer.common_name View Source
sentinel_one activity os.family View Source
sentinel_one alert os.name View Source
trend_micro_vision_one detection os.name View Source

(List generated with an agg on top of query @attributes.deprecated:false and @attributes.rsa2elk:false and @owner:elastic/security-external-integrations and @type:field and name:(vlan.id or geo.continent_name or os.type or interface.id or os.name or interface.name or as.number or os.name or os.name or as.number or os.family or os.type or interface.name or x509.issuer.common_name or geo.city_name) to https://github.com/andrewkroh/go-examples/tree/main/fleetpkg-indexer)

elasticmachine commented 10 months ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kgeller commented 9 months ago

Just for my own record keeping sake I am converting the above table into categories: 1) in need of pipeline updates 2) false alarms that only need cleanup to their ecs field declarations 3) omitted

Integrations in need of pipeline updates:

Integrations for ecs field declaration updates:

Omitted

kgeller commented 9 months ago

All non-deprecated packages noted above have now been updated.

andrewkroh commented 9 months ago

juniper_srx and netflow still have issues.

packages/juniper_srx/data_stream/log/fields/ecs.yml:15:3 as.organization.name
packages/juniper_srx/data_stream/log/fields/ecs.yml:81:3 code_signature.exists
packages/juniper_srx/data_stream/log/fields/ecs.yml:83:3 code_signature.status
packages/juniper_srx/data_stream/log/fields/ecs.yml:85:3 code_signature.subject_name
packages/juniper_srx/data_stream/log/fields/ecs.yml:87:3 code_signature.trusted
packages/juniper_srx/data_stream/log/fields/ecs.yml:89:3 code_signature.valid
packages/juniper_srx/data_stream/log/fields/ecs.yml:367:3 hash.md5
packages/juniper_srx/data_stream/log/fields/ecs.yml:369:3 hash.sha1
packages/juniper_srx/data_stream/log/fields/ecs.yml:371:3 hash.sha256
packages/juniper_srx/data_stream/log/fields/ecs.yml:373:3 hash.sha512
packages/juniper_srx/data_stream/log/fields/ecs.yml:555:3 pe.architecture
packages/juniper_srx/data_stream/log/fields/ecs.yml:557:3 pe.company
packages/juniper_srx/data_stream/log/fields/ecs.yml:559:3 pe.description
packages/juniper_srx/data_stream/log/fields/ecs.yml:561:3 pe.file_version
packages/juniper_srx/data_stream/log/fields/ecs.yml:563:3 pe.imphash
packages/juniper_srx/data_stream/log/fields/ecs.yml:565:3 pe.original_file_name
packages/juniper_srx/data_stream/log/fields/ecs.yml:567:3 pe.product
packages/netflow/data_stream/log/fields/ecs.yml:13:3 as.organization.name
packages/netflow/data_stream/log/fields/ecs.yml:283:3 geo.city_name
packages/netflow/data_stream/log/fields/ecs.yml:285:3 geo.continent_name
packages/netflow/data_stream/log/fields/ecs.yml:287:3 geo.country_iso_code
packages/netflow/data_stream/log/fields/ecs.yml:289:3 geo.country_name
packages/netflow/data_stream/log/fields/ecs.yml:291:3 geo.location
packages/netflow/data_stream/log/fields/ecs.yml:293:3 geo.name
packages/netflow/data_stream/log/fields/ecs.yml:295:3 geo.region_iso_code
packages/netflow/data_stream/log/fields/ecs.yml:297:3 geo.region_name
packages/netflow/data_stream/log/fields/ecs.yml:305:3 hash.md5
packages/netflow/data_stream/log/fields/ecs.yml:307:3 hash.sha1
packages/netflow/data_stream/log/fields/ecs.yml:309:3 hash.sha256
packages/netflow/data_stream/log/fields/ecs.yml:311:3 hash.sha512
packages/netflow/data_stream/log/fields/ecs.yml:467:3 os.family
packages/netflow/data_stream/log/fields/ecs.yml:469:3 os.full
packages/netflow/data_stream/log/fields/ecs.yml:471:3 os.kernel
packages/netflow/data_stream/log/fields/ecs.yml:473:3 os.name
packages/netflow/data_stream/log/fields/ecs.yml:475:3 os.platform
packages/netflow/data_stream/log/fields/ecs.yml:477:3 os.version