[Network Packet Capture]: No more GeoIP resolution and event.dataset missing

[Danouchka]

Integration Name

Network Packet Capture [network_traffic]

Dataset Name

event.dataset is missing

Integration Version

1.32.0

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Elasticsearch in Elastic Cloud. Agents are on Centos7 and Debian 11

Software/API Version

No response

Error Message

Since I have upgraded the integration to 1.32.0 on Friday August 30th, I see that records produced by this integration does not have the event.dataset field and GeoIP resolution for source.ip and destination.ip is missing (specifically for records related to data_stream.dataset :"network_traffic.flow" )

Event Original

Exemple of produced record

{ "_index": ".ds-logs-network_traffic.flow-default-2024.09.02-000146", "_id": "Xaq-sZEBaOOMdreY2xo3", "_version": 1, "_score": 0, "_source": { "agent": { "name": "sa-da-ingest-02", "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "ephemeral_id": "bfa71e54-fc46-44b0-aff5-1bc570126678", "type": "packetbeat", "version": "8.15.0" }, "elastic_agent": { "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "version": "8.15.0", "snapshot": false }, "destination": { "port": 53, "bytes": 241, "ip": "169.254.169.254", "packets": 1 }, "network_traffic": { "flow": { "final": true, "id": "EAL/////AP////8I//8AAAEKhA/Iqf6p/milNQA" } }, "source": { "port": 42344, "bytes": 134, "ip": "10.132.15.200", "packets": 1 }, "network": { "community_id": "1:3XiPXM8UYTAheqHBrxSYIMvtDEM=", "bytes": 375, "transport": "udp", "type": "ipv4", "packets": 2 }, "cloud": { "availability_zone": "europe-west1-b", "instance": { "name": "sa-da-ingest-02", "id": "2122642883198246451" }, "provider": "gcp", "service": { "name": "GCE" }, "machine": { "type": "c2-standard-4" }, "project": { "id": "elastic-sa" }, "region": "europe-west1", "account": { "id": "elastic-sa" } }, "@timestamp": "2024-09-02T07:59:20.001Z", "ecs": { "version": "8.11.0" }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "network_traffic.flow" }, "host": { "hostname": "sa-da-ingest-02", "os": { "kernel": "3.10.0-1160.15.2.el7.x86_64", "codename": "Core", "name": "CentOS Linux", "type": "linux", "family": "redhat", "version": "7 (Core)", "platform": "centos" }, "containerized": false, "ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "name": "sa-da-ingest-02", "id": "012a787168254cbcaa5f13dde54611bc", "mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "architecture": "x86_64" }, "event": { "duration": 18586333, "agent_id_status": "verified", "ingested": "2024-09-02T07:59:20Z", "kind": "event", "start": "2024-09-02T07:58:26.259Z", "action": "network_flow", "end": "2024-09-02T07:58:26.277Z", "category": [ "network" ], "type": [ "connection", "end" ] } }, "fields": { "elastic_agent.version": [ "8.15.0" ], "event.category": [ "network" ], "host.os.name.text": [ "CentOS Linux" ], "host.hostname": [ "sa-da-ingest-02" ], "host.mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "cloud.availability_zone": [ "europe-west1-b" ], "host.os.version": [ "7 (Core)" ], "host.os.name": [ "CentOS Linux" ], "source.ip": [ "10.132.15.200" ], "agent.name": [ "sa-da-ingest-02" ], "host.name": [ "sa-da-ingest-02" ], "network.community_id": [ "1:3XiPXM8UYTAheqHBrxSYIMvtDEM=" ], "event.agent_id_status": [ "verified" ], "event.kind": [ "event" ], "source.packets": [ 1 ], "cloud.region": [ "europe-west1" ], "host.os.type": [ "linux" ], "network.packets": [ 2 ], "data_stream.type": [ "logs" ], "host.architecture": [ "x86_64" ], "cloud.machine.type": [ "c2-standard-4" ], "cloud.provider": [ "gcp" ], "agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "cloud.service.name": [ "GCE" ], "source.port": [ 42344 ], "ecs.version": [ "8.11.0" ], "host.containerized": [ false ], "agent.version": [ "8.15.0" ], "destination.bytes": [ 241 ], "event.start": [ "2024-09-02T07:58:26.259Z" ], "host.os.family": [ "redhat" ], "destination.port": [ 53 ], "event.end": [ "2024-09-02T07:58:26.277Z" ], "destination.packets": [ 1 ], "cloud.instance.id": [ "2122642883198246451" ], "host.ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "agent.type": [ "packetbeat" ], "host.os.kernel": [ "3.10.0-1160.15.2.el7.x86_64" ], "network.bytes": [ 375 ], "elastic_agent.snapshot": [ false ], "host.id": [ "012a787168254cbcaa5f13dde54611bc" ], "network.type": [ "ipv4" ], "source.bytes": [ 134 ], "network_traffic.flow.final": [ true ], "elastic_agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "data_stream.namespace": [ "default" ], "host.os.codename": [ "Core" ], "destination.ip": [ "169.254.169.254" ], "network_traffic.flow.id": [ "EAL/////AP////8I//8AAAEKhA/Iqf6p/milNQA" ], "network.transport": [ "udp" ], "event.duration": [ 18586333 ], "event.action": [ "network_flow" ], "event.ingested": [ "2024-09-02T07:59:20.000Z" ], "@timestamp": [ "2024-09-02T07:59:20.001Z" ], "cloud.account.id": [ "elastic-sa" ], "host.os.platform": [ "centos" ], "data_stream.dataset": [ "network_traffic.flow" ], "event.type": [ "connection", "end" ], "agent.ephemeral_id": [ "bfa71e54-fc46-44b0-aff5-1bc570126678" ], "cloud.instance.name": [ "sa-da-ingest-02" ], "cloud.project.id": [ "elastic-sa" ] } }

What did you do?

Nothing special , just upgraded the integration

What did you see?

event.dataset is missing no GeoIP resolution for source.ip and destination.ip as it used to be

What did you expect to see?

I expect to have the field event.dataset properly filled and source.ip and destination.ip to be geoIP resolved

ex of records before it was broken

{ "_index": ".ds-logs-network_traffic.flow-default-2024.08.26-000145", "id": "grBJo5EB7Gj699C53T1", "_version": 1, "_score": 0, "_source": { "process": { "args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "start": "2024-08-18T10:50:02.480Z", "name": "elastic-agent", "working_directory": "", "pid": 20251, "executable": "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent", "ppid": 1 }, "agent": { "name": "sa-da-ingest-02", "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "type": "packetbeat", "ephemeral_id": "bfa71e54-fc46-44b0-aff5-1bc570126678", "version": "8.15.0" }, "destination": { "process": { "args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "name": "elastic-agent", "start": "2024-08-18T10:50:02.480Z", "working_directory": "", "pid": 20251, "executable": "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent", "ppid": 1 }, "port": 43054, "bytes": 1161, "ip": "10.132.15.200", "packets": 5 }, "elastic_agent": { "id": "bcfa6932-65a3-4872-8727-e2adbc1a4920", "version": "8.15.0", "snapshot": false }, "source": { "geo": { "continent_name": "Europe", "region_iso_code": "BE-BRU", "city_name": "Brussels", "country_iso_code": "BE", "country_name": "Belgium", "location": { "lon": 4.347, "lat": 50.8534 }, "region_name": "Brussels Capital" }, "as": { "number": 396982, "organization": { "name": "GOOGLE-CLOUD-PLATFORM" } }, "port": 443, "bytes": 9498, "ip": "35.195.130.253", "packets": 5 }, "type": "flow", "network": { "community_id": "1:tkaSkzYIv7W9j51IIb6czxEDI6k=", "bytes": 10659, "transport": "tcp", "type": "ipv4", "packets": 10 }, "cloud": { "availability_zone": "europe-west1-b", "instance": { "name": "sa-da-ingest-02", "id": "2122642883198246451" }, "provider": "gcp", "machine": { "type": "c2-standard-4" }, "service": { "name": "GCE" }, "project": { "id": "elastic-sa" }, "region": "europe-west1", "account": { "id": "elastic-sa" } }, "@timestamp": "2024-08-30T12:36:43.192Z", "ecs": { "version": "8.11.0" }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "network_traffic.flow" }, "host": { "hostname": "sa-da-ingest-02", "os": { "kernel": "3.10.0-1160.15.2.el7.x86_64", "codename": "Core", "name": "CentOS Linux", "family": "redhat", "type": "linux", "version": "7 (Core)", "platform": "centos" }, "containerized": false, "ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "name": "sa-da-ingest-02", "id": "012a787168254cbcaa5f13dde54611bc", "mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "architecture": "x86_64" }, "event": { "duration": 1183622761, "agent_id_status": "verified", "ingested": "2024-08-30T12:36:52Z", "kind": "event", "start": "2024-08-30T12:36:41.997Z", "action": "network_flow", "end": "2024-08-30T12:36:43.181Z", "type": [ "connection", "end" ], "category": [ "network" ], "dataset": "network_traffic.flow" }, "flow": { "final": true, "id": "EAz/////AP//////CAwAAAEKhA/II8OC/S6ouwEiAAAAAAAAAA" } }, "fields": { "flow.id": [ "EAz/////AP//////CAwAAAEKhA/II8OC/S6ouwEiAAAAAAAAAA" ], "elastic_agent.version": [ "8.15.0" ], "event.category": [ "network" ], "process.name.text": [ "elastic-agent" ], "host.os.name.text": [ "CentOS Linux" ], "host.hostname": [ "sa-da-ingest-02" ], "process.pid": [ 20251 ], "type": [ "flow" ], "host.mac": [ "02-42-F0-1E-10-90", "42-01-0A-84-0F-C8" ], "cloud.availability_zone": [ "europe-west1-b" ], "host.os.version": [ "7 (Core)" ], "destination.process.args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "source.geo.region_name": [ "Brussels Capital" ], "host.os.name": [ "CentOS Linux" ], "source.ip": [ "35.195.130.253" ], "agent.name": [ "sa-da-ingest-02" ], "host.name": [ "sa-da-ingest-02" ], "network.community_id": [ "1:tkaSkzYIv7W9j51IIb6czxEDI6k=" ], "event.agent_id_status": [ "verified" ], "source.geo.region_iso_code": [ "BE-BRU" ], "event.kind": [ "event" ], "source.geo.city_name": [ "Brussels" ], "flow.final": [ true ], "source.packets": [ 5 ], "cloud.region": [ "europe-west1" ], "host.os.type": [ "linux" ], "network.packets": [ 10 ], "process.ppid": [ 1 ], "destination.process.name.text": [ "elastic-agent" ], "data_stream.type": [ "logs" ], "host.architecture": [ "x86_64" ], "process.name": [ "elastic-agent" ], "cloud.machine.type": [ "c2-standard-4" ], "cloud.provider": [ "gcp" ], "agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "cloud.service.name": [ "GCE" ], "source.port": [ 443 ], "ecs.version": [ "8.11.0" ], "host.containerized": [ false ], "destination.process.start": [ "2024-08-18T10:50:02.480Z" ], "agent.version": [ "8.15.0" ], "destination.process.pid": [ 20251 ], "destination.bytes": [ 1161 ], "event.start": [ "2024-08-30T12:36:41.997Z" ], "host.os.family": [ "redhat" ], "source.as.number": [ 396982 ], "process.start": [ "2024-08-18T10:50:02.480Z" ], "destination.port": [ 43054 ], "destination.process.name": [ "elastic-agent" ], "event.end": [ "2024-08-30T12:36:43.181Z" ], "destination.process.ppid": [ 1 ], "source.geo.location": [ { "coordinates": [ 4.347, 50.8534 ], "type": "Point" } ], "process.working_directory": [ "" ], "destination.packets": [ 5 ], "destination.process.executable": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "cloud.instance.id": [ "2122642883198246451" ], "host.ip": [ "10.132.15.200", "fe80::d83d:fb17:cc5c:2b56", "172.17.0.1", "fe80::42:f0ff:fe1e:1090" ], "agent.type": [ "packetbeat" ], "process.executable.text": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "destination.process.executable.text": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "host.os.kernel": [ "3.10.0-1160.15.2.el7.x86_64" ], "source.geo.country_iso_code": [ "BE" ], "network.bytes": [ 10659 ], "elastic_agent.snapshot": [ false ], "destination.process.working_directory.text": [ "" ], "host.id": [ "012a787168254cbcaa5f13dde54611bc" ], "network.type": [ "ipv4" ], "process.executable": [ "/var/lib/elastic-agent/data/elastic-agent-8.15.0-25075f/elastic-agent" ], "source.bytes": [ 9498 ], "destination.process.working_directory": [ "" ], "source.as.organization.name.text": [ "GOOGLE-CLOUD-PLATFORM" ], "elastic_agent.id": [ "bcfa6932-65a3-4872-8727-e2adbc1a4920" ], "data_stream.namespace": [ "default" ], "process.working_directory.text": [ "" ], "host.os.codename": [ "Core" ], "process.args": [ "/usr/share/elastic-agent/bin/elastic-agent", "--path.home", "/var/lib/elastic-agent", "--path.config", "/etc/elastic-agent", "--path.logs", "/var/log/elastic-agent", "run", "--environment", "systemd", "-c", "/etc/elastic-agent/elastic-agent.yml" ], "source.as.organization.name": [ "GOOGLE-CLOUD-PLATFORM" ], "source.geo.continent_name": [ "Europe" ], "destination.ip": [ "10.132.15.200" ], "network.transport": [ "tcp" ], "event.duration": [ 1183622761 ], "event.action": [ "network_flow" ], "event.ingested": [ "2024-08-30T12:36:52.000Z" ], "@timestamp": [ "2024-08-30T12:36:43.192Z" ], "cloud.account.id": [ "elastic-sa" ], "host.os.platform": [ "centos" ], "data_stream.dataset": [ "network_traffic.flow" ], "event.type": [ "connection", "end" ], "agent.ephemeral_id": [ "bfa71e54-fc46-44b0-aff5-1bc570126678" ], "source.geo.country_name": [ "Belgium" ], "event.dataset": [ "network_traffic.flow" ], "cloud.instance.name": [ "sa-da-ingest-02" ], "cloud.project.id": [ "elastic-sa" ] } }

Anything else?

No response

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[efd6]

The removal of event.dataset is part of https://github.com/elastic/integrations/issues/8185 and I suspect that the geoip issue is due to configuration since this is configurable.

[Danouchka]

The issue is that event.dataset is used as a partition field in the default ML jobs for Log Categorization in Observability The fact that it is disappearing may be an issue

[Danouchka]

For the geoip configuration issue, seems the integration upgrade miss the configuration value. I reenabled the toggle (switch off et switch on again ) and this works again.

[Danouchka]

But the event.dataset should be kept for the reasons I have mentionned above

[efd6]

The change is in line with the planned deprecation and retirement plan in #8185 and is stage 3. (perhaps with one exception; I'm not sure if the fleet changes that are indicated there to have happened actually did /cc @nimarezainia). I've marked the PR into the plan, but left it as not completed. The removal of event.dataset should not prevent you from partitioning the documents; data_stream.dataset provide the same information, which is why event.dataset was removed.

If you absolutely need to use event.dataset in the short term, you can change the configuration for "Map root Packetbeat fields to ECS" to false, though note that the option is planned to be removed completely in six months.

[nimarezainia]

@efd6 no unfortunately I missed this ping. I don;t think we made any changes in the integrations/fleet in this regard. will follow up in the original issue.

[Danouchka]

Thank you ! I think we can close the ticket. Just to be aware, might switch off and on again the GeoIP resolution configuration toggle