search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
27 stars 444 forks source link

[Network Packet Capture]: No more GeoIP resolution and event.dataset missing #10956

Closed Danouchka closed 2 months ago

Danouchka commented 2 months ago

Integration Name

Network Packet Capture [network_traffic]

Dataset Name

event.dataset is missing

Integration Version

1.32.0

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Elasticsearch in Elastic Cloud. Agents are on Centos7 and Debian 11

Software/API Version

No response

Error Message

Since I have upgraded the integration to 1.32.0 on Friday August 30th, I see that records produced by this integration does not have the event.dataset field and GeoIP resolution for source.ip and destination.ip is missing (specifically for records related to data_stream.dataset :"network_traffic.flow" )

Event Original

Exemple of produced record

{ "_index": ".ds-logs-network_traffic.flow-default-2024.09.02-000146", "_id": "Xaq-sZEBaOOMdreY2xo3", "_version": 1, "_score": 0, "_source": { "agent": { "name": "sa-da-ingest-02", "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "ephemeral_id": "bfa71e54-fc46-44b0-aff5-1bc570126678", "type": "packetbeat", "version": "8.15.0" }, "elastic_agent": { "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "version": "8.15.0", "snapshot": false }, "destination": { "port": 53, "bytes": 241, "ip": "169.254.169.254", "packets": 1 }, "network_traffic": { "flow": { "final": true, "id": "EAL/////AP////8I//8AAAEKhA/Iqf6p/milNQA" } }, "source": { "port": 42344, "bytes": 134, "ip": "10.132.15.200", "packets": 1 }, "network": { "community_id": "1:3XiPXM8UYTAheqHBrxSYIMvtDEM=", "bytes": 375, "transport": "udp", "type": "ipv4", "packets": 2 }, "cloud": { "availability_zone": "europe-west1-b", "instance": { "name": "sa-da-ingest-02", "id": "2122642883198246451" }, "provider": "gcp", "service": { "name": "GCE" }, "machine": { "type": "c2-standard-4" }, "project": { "id": "elastic-sa" }, "region": "europe-west1", "account": { "id": "elastic-sa" } }, "@timestamp": "2024-09-02T07:59:20.001Z", "ecs": { "version": "8.11.0" }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "network_traffic.flow" }, "host": { "hostname": "sa-da-ingest-02", "os": { "kernel": "3.10.0-1160.15.2.el7.x86_64", "codename": "Core", "name": "CentOS Linux", "type": "linux", "family": "redhat", "version": "7 (Core)", "platform": "centos" }, "containerized": false, "ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "name": "sa-da-ingest-02", "id": "012a787168254cbcaa5f13dde54611bc", "mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "architecture": "x86_64" }, "event": { "duration": 18586333, "agent_id_status": "verified", "ingested": "2024-09-02T07:59:20Z", "kind": "event", "start": "2024-09-02T07:58:26.259Z", "action": "network_flow", "end": "2024-09-02T07:58:26.277Z", "category": [ "network" ], "type": [ "connection", "end" ] } }, "fields": { "elastic_agent.version": [ "8.15.0" ], "event.category": [ "network" ], "host.os.name.text": [ "CentOS Linux" ], "host.hostname": [ "sa-da-ingest-02" ], "host.mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "cloud.availability_zone": [ "europe-west1-b" ], "host.os.version": [ "7 (Core)" ], "host.os.name": [ "CentOS Linux" ], "source.ip": [ "10.132.15.200" ], "agent.name": [ "sa-da-ingest-02" ], "host.name": [ "sa-da-ingest-02" ], "network.community_id": [ "1:3XiPXM8UYTAheqHBrxSYIMvtDEM=" ], "event.agent_id_status": [ "verified" ], "event.kind": [ "event" ], "source.packets": [ 1 ], "cloud.region": [ "europe-west1" ], "host.os.type": [ "linux" ], "network.packets": [ 2 ], "data_stream.type": [ "logs" ], "host.architecture": [ "x86_64" ], "cloud.machine.type": [ "c2-standard-4" ], "cloud.provider": [ "gcp" ], "agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "cloud.service.name": [ "GCE" ], "source.port": [ 42344 ], "ecs.version": [ "8.11.0" ], "host.containerized": [ false ], "agent.version": [ "8.15.0" ], "destination.bytes": [ 241 ], "event.start": [ "2024-09-02T07:58:26.259Z" ], "host.os.family": [ "redhat" ], "destination.port": [ 53 ], "event.end": [ "2024-09-02T07:58:26.277Z" ], "destination.packets": [ 1 ], "cloud.instance.id": [ "2122642883198246451" ], "host.ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "agent.type": [ "packetbeat" ], "host.os.kernel": [ "3.10.0-1160.15.2.el7.x86_64" ], "network.bytes": [ 375 ], "elastic_agent.snapshot": [ false ], "host.id": [ "012a787168254cbcaa5f13dde54611bc" ], "network.type": [ "ipv4" ], "source.bytes": [ 134 ], "network_traffic.flow.final": [ true ], "elastic_agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "data_stream.namespace": [ "default" ], "host.os.codename": [ "Core" ], "destination.ip": [ "169.254.169.254" ], "network_traffic.flow.id": [ "EAL/////AP////8I//8AAAEKhA/Iqf6p/milNQA" ], "network.transport": [ "udp" ], "event.duration": [ 18586333 ], "event.action": [ "network_flow" ], "event.ingested": [ "2024-09-02T07:59:20.000Z" ], "@timestamp": [ "2024-09-02T07:59:20.001Z" ], "cloud.account.id": [ "elastic-sa" ], "host.os.platform": [ "centos" ], "data_stream.dataset": [ "network_traffic.flow" ], "event.type": [ "connection", "end" ], "agent.ephemeral_id": [ "bfa71e54-fc46-44b0-aff5-1bc570126678" ], "cloud.instance.name": [ "sa-da-ingest-02" ], "cloud.project.id": [ "elastic-sa" ] } }

What did you do?

Nothing special , just upgraded the integration

What did you see?

event.dataset is missing no GeoIP resolution for source.ip and destination.ip as it used to be

What did you expect to see?

I expect to have the field event.dataset properly filled and source.ip and destination.ip to be geoIP resolved

ex of records before it was broken

{ "_index": ".ds-logs-network_traffic.flow-default-2024.08.26-000145", "id": "grBJo5EB7Gj699C53T1", "_version": 1, "_score": 0, "_source": { "process": { "args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "start": "2024-08-18T10:50:02.480Z", "name": "elastic-agent", "working_directory": "", "pid": 20251, "executable": "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent", "ppid": 1 }, "agent": { "name": "sa-da-ingest-02", "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "type": "packetbeat", "ephemeral_id": "bfa71e54-fc46-44b0-aff5-1bc570126678", "version": "8.15.0" }, "destination": { "process": { "args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "name": "elastic-agent", "start": "2024-08-18T10:50:02.480Z", "working_directory": "", "pid": 20251, "executable": "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent", "ppid": 1 }, "port": 43054, "bytes": 1161, "ip": "10.132.15.200", "packets": 5 }, "elastic_agent": { "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "version": "8.15.0", "snapshot": false }, "source": { "geo": { "continent_name": "Europe", "region_iso_code": "BE-BRU", "city_name": "Brussels", "country_iso_code": "BE", "country_name": "Belgium", "location": { "lon": 4.347, "lat": 50.8534 }, "region_name": "Brussels Capital" }, "as": { "number": 396982, "organization": { "name": "GOOGLE-CLOUD-PLATFORM" } }, "port": 443, "bytes": 9498, "ip": "35.195.130.253", "packets": 5 }, "type": "flow", "network": { "community_id": "1:tkaSkzYIv7W9j51IIb6czxEDI6k=", "bytes": 10659, "transport": "tcp", "type": "ipv4", "packets": 10 }, "cloud": { "availability_zone": "europe-west1-b", "instance": { "name": "sa-da-ingest-02", "id": "2122642883198246451" }, "provider": "gcp", "machine": { "type": "c2-standard-4" }, "service": { "name": "GCE" }, "project": { "id": "elastic-sa" }, "region": "europe-west1", "account": { "id": "elastic-sa" } }, "@timestamp": "2024-08-30T12:36:43.192Z", "ecs": { "version": "8.11.0" }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "network_traffic.flow" }, "host": { "hostname": "sa-da-ingest-02", "os": { "kernel": "3.10.0-1160.15.2.el7.x86_64", "codename": "Core", "name": "CentOS Linux", "family": "redhat", "type": "linux", "version": "7 (Core)", "platform": "centos" }, "containerized": false, "ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "name": "sa-da-ingest-02", "id": "012a787168254cbcaa5f13dde54611bc", "mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "architecture": "x86_64" }, "event": { "duration": 1183622761, "agent_id_status": "verified", "ingested": "2024-08-30T12:36:52Z", "kind": "event", "start": "2024-08-30T12:36:41.997Z", "action": "network_flow", "end": "2024-08-30T12:36:43.181Z", "type": [ "connection", "end" ], "category": [ "network" ], "dataset": "network_traffic.flow" }, "flow": { "final": true, "id": "EAz/////AP//////CAwAAAEKhA/II8OC/S6ouwEiAAAAAAAAAA" } }, "fields": { "flow.id": [ "EAz/////AP//////CAwAAAEKhA/II8OC/S6ouwEiAAAAAAAAAA" ], "elastic_agent.version": [ "8.15.0" ], "event.category": [ "network" ], "process.name.text": [ "elastic-agent" ], "host.os.name.text": [ "CentOS Linux" ], "host.hostname": [ "sa-da-ingest-02" ], "process.pid": [ 20251 ], "type": [ "flow" ], "host.mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "cloud.availability_zone": [ "europe-west1-b" ], "host.os.version": [ "7 (Core)" ], "destination.process.args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "source.geo.region_name": [ "Brussels Capital" ], "host.os.name": [ "CentOS Linux" ], "source.ip": [ "35.195.130.253" ], "agent.name": [ "sa-da-ingest-02" ], "host.name": [ "sa-da-ingest-02" ], "network.community_id": [ "1:tkaSkzYIv7W9j51IIb6czxEDI6k=" ], "event.agent_id_status": [ "verified" ], "source.geo.region_iso_code": [ "BE-BRU" ], "event.kind": [ "event" ], "source.geo.city_name": [ "Brussels" ], "flow.final": [ true ], "source.packets": [ 5 ], "cloud.region": [ "europe-west1" ], "host.os.type": [ "linux" ], "network.packets": [ 10 ], "process.ppid": [ 1 ], "destination.process.name.text": [ "elastic-agent" ], "data_stream.type": [ "logs" ], "host.architecture": [ "x86_64" ], "process.name": [ "elastic-agent" ], "cloud.machine.type": [ "c2-standard-4" ], "cloud.provider": [ "gcp" ], "agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "cloud.service.name": [ "GCE" ], "source.port": [ 443 ], "ecs.version": [ "8.11.0" ], "host.containerized": [ false ], "destination.process.start": [ "2024-08-18T10:50:02.480Z" ], "agent.version": [ "8.15.0" ], "destination.process.pid": [ 20251 ], "destination.bytes": [ 1161 ], "event.start": [ "2024-08-30T12:36:41.997Z" ], "host.os.family": [ "redhat" ], "source.as.number": [ 396982 ], "process.start": [ "2024-08-18T10:50:02.480Z" ], "destination.port": [ 43054 ], "destination.process.name": [ "elastic-agent" ], "event.end": [ "2024-08-30T12:36:43.181Z" ], "destination.process.ppid": [ 1 ], "source.geo.location": [ { "coordinates": [ 4.347, 50.8534 ], "type": "Point" } ], "process.working_directory": [ "" ], "destination.packets": [ 5 ], "destination.process.executable": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "cloud.instance.id": [ "2122642883198246451" ], "host.ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "agent.type": [ "packetbeat" ], "process.executable.text": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "destination.process.executable.text": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "host.os.kernel": [ "3.10.0-1160.15.2.el7.x86_64" ], "source.geo.country_iso_code": [ "BE" ], "network.bytes": [ 10659 ], "elastic_agent.snapshot": [ false ], "destination.process.working_directory.text": [ "" ], "host.id": [ "012a787168254cbcaa5f13dde54611bc" ], "network.type": [ "ipv4" ], "process.executable": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "source.bytes": [ 9498 ], "destination.process.working_directory": [ "" ], "source.as.organization.name.text": [ "GOOGLE-CLOUD-PLATFORM" ], "elastic_agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "data_stream.namespace": [ "default" ], "process.working_directory.text": [ "" ], "host.os.codename": [ "Core" ], "process.args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "source.as.organization.name": [ "GOOGLE-CLOUD-PLATFORM" ], "source.geo.continent_name": [ "Europe" ], "destination.ip": [ "10.132.15.200" ], "network.transport": [ "tcp" ], "event.duration": [ 1183622761 ], "event.action": [ "network_flow" ], "event.ingested": [ "2024-08-30T12:36:52.000Z" ], "@timestamp": [ "2024-08-30T12:36:43.192Z" ], "cloud.account.id": [ "elastic-sa" ], "host.os.platform": [ "centos" ], "data_stream.dataset": [ "network_traffic.flow" ], "event.type": [ "connection", "end" ], "agent.ephemeral_id": [ "bfa71e54-fc46-44b0-aff5-1bc570126678" ], "source.geo.country_name": [ "Belgium" ], "event.dataset": [ "network_traffic.flow" ], "cloud.instance.name": [ "sa-da-ingest-02" ], "cloud.project.id": [ "elastic-sa" ] } }

Anything else?

No response

elasticmachine commented 2 months ago

Pinging @elastic/integrations (Team:Integrations)

elasticmachine commented 2 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elasticmachine commented 2 months ago

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

efd6 commented 2 months ago

The removal of event.dataset is part of https://github.com/elastic/integrations/issues/8185 and I suspect that the geoip issue is due to configuration since this is configurable.

Danouchka commented 2 months ago

The issue is that event.dataset is used as a partition field in the default ML jobs for Log Categorization in Observability The fact that it is disappearing may be an issue

Danouchka commented 2 months ago

For the geoip configuration issue, seems the integration upgrade miss the configuration value. I reenabled the toggle (switch off et switch on again ) and this works again.

Danouchka commented 2 months ago

But the event.dataset should be kept for the reasons I have mentionned above

efd6 commented 2 months ago

The change is in line with the planned deprecation and retirement plan in #8185 and is stage 3. (perhaps with one exception; I'm not sure if the fleet changes that are indicated there to have happened actually did /cc @nimarezainia). I've marked the PR into the plan, but left it as not completed. The removal of event.dataset should not prevent you from partitioning the documents; data_stream.dataset provide the same information, which is why event.dataset was removed.

If you absolutely need to use event.dataset in the short term, you can change the configuration for "Map root Packetbeat fields to ECS" to false, though note that the option is planned to be removed completely in six months.

nimarezainia commented 2 months ago

@efd6 no unfortunately I missed this ping. I don;t think we made any changes in the integrations/fleet in this regard. will follow up in the original issue.

Danouchka commented 2 months ago

Thank you ! I think we can close the ticket. Just to be aware, might switch off and on again the GeoIP resolution configuration toggle