Add network.address

[vbohata]

I often need to somehow store the network name and network address. I propose to add network.address field for values like 192.168.0.0/24 ...

[ruflin]

@webmat @robgil Would be great to get your feedback on this one.

@vbohata Can you share a bit of background on what queries you would run against the above field?

[vbohata]

Currently for firewall logs I filter documents by exact match of some network address value like "192.168.50.0/24". It would be handy to be able to do some network vs. ip comparison natively in ES, but ES do not have a data type for network address.

[robgil]

@ruflin @vbohata Yes, this makes sense. We definitely need a field for this. How about network.cidr? This would avoid confusion with other addresses like MAC or IP. We should probably also add network.netmask to be complete here.

[vbohata]

network.cidr sounds reasonable. Netmask is also useful, but if there will be netmask, you should add also network base address. The network object will be maybe even more complicated because I can imagine someone will need to store both IPv4 and IPv6 addresses. Also there is missing interface object (maybe network.interface.name) which will be probably handy for metricbeat network module.

[ruflin]

++ on cidr. Do you want to open a PR?

[vbohata]

Yes, please.

[vbohata]

Btw. I just found ES knows ip_range type which can be entered as CIDR ... good to know :)

[ruflin]

Good point, should we make the field an ip_range then?

Also nice to see that ip addresses can be queried with a cidr notation: https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html

[vbohata]

AFAIK for term aggregations it should be keyword, for ip comparisons it should be ip_range. So probably network.cidr should be ip_range type and using fields network.cidr.keyword for keyword data type.

[MikePaquette]

@vbohata @rgil Quick question: From where would this field be populated? For example is this a field that would normally be found in an event? Can you provide examples? Or is this a kind of enrichment field that you might populate by doing a lookup from an external db?

[robgil]

@MikePaquette any number of logs will write cidr ranges. @vbohata mentions their firewall logs above, but this can also be BGP logs, firewall policy logs (when policies apply to cidr ranges), VPN/IPSec Logs (left / right networks are in cidr notation), VPCs, VPC subnets (source and destination subnet for example). Another even more common case would be to log the CIDR of a public IP thats involved in an attack. You might log the CIDR, owner, country of origin, etc of an attacking/suspicious IP.

This can also be used for enrichment. Examples

• Adding the cidr a particular host is part of 10.10.10.10 being in 10.10.10.0/24
• VPC specific events (cloudtrail logs can be enriched with the CIDR of the VPC)

[vbohata]

Also part of dhcp logs and more.

[MikePaquette]

Thanks @robgil @vbohata Sounds good.

[ruflin]

@robgil Do you want to open a PR with it?

[dainperkins]

Isn't CIDR and subnet repetitious? (CIDR includes mask, even if its in # of bits)

[robgil]

@dainperkins to a degree yes, but when it comes to searching, it will be useful to have the full expansion.

[dainperkins]

@robgil Just wondering - penalty for computing on ingest being less then searching on combined fields? Wondering how to get from any likely network log source or destination Ip field -> CIDR/subnet - seems like the kind of thing that would be more applicable to pull from hosts, or a compare against a fixed list of internal subnets. (e.g. host.subnet / host.cidr)

I still have a tendency to look at big data from a network/security guys hack understanding of e.g. 3rd normal form, and i don't think I've quite wrapped my head around the overall ECS schema architecture (e.g. host.ip vs. source.ip/destination.ip)

[robgil]

@dainperkins for me the intent of the additional fields is to allow operators to map their logs to a common schema. Having the fields available gives operators the option in how they want to search their logs. For example, a CIDR is not the same as a netmask. CIDR is a combined field of both the network and subnet mask. Having the option to have these in their requisite base fields (network and subnet netmask) would allow operators to search based on how their workflow dictates and how their logs present the information.

For example, how would someone search for all /24 networks (255.255.255.0)? If it was only a cidr, you'd need to specify a valid cidr range.. I don't think you could search just "*/24" for example. So if someone has logs that write in the format network (10.10.10.10), netmask (255.255.255.0), the operator would need to convert it to CIDR notation vs. just mapping the network and netmask. Does that make sense? I was trying to make it as flexible as possible.

[dainperkins]

@robgil Thanks, yeah I think I figured out where I was getting wrapped round the axle...

for any given alert, from any given system, pull the appropriate field names from any of the ECS scope (important piece is use appropriate fields given the source information, as opposed to a global idea of relevancy / normalization, etc.)

e.g. Firewall might use source.ip, source.port, destination.ip, destination.port, as well as tls.version, etc. EDR/Vulnerability might use host.ip, as theres no concept of source/destination

to go back to CIDR - that'll never come in from anything reporting at a network level on the far side of a router, but e.g. CarbonBlack or AMP may well provide that information - which can then be utilized for searching / aggregating etc.

I was thinking more form a join/disparate event type correlation standpoint - e.g. if i want to search every event in elastic for events regarding ip addresses in a given class C (192.168.1.0/24) CIDR and netmask are unlikely to help for firewall logs unless I have some sort of mapping from IP to subnet that would allow me to populate e.g firewall logs with source.network when a 192.168.1.23 source.ip address is seen, or a search which understands that any in 192.168.1.0/24 is the same as e.g a regex for 192.168.1.[0-255])

[mdelapenya]

As @vbohata mentioned above, I also miss a network.interface.name field. I.e. for firewall logs it is very common to have something like this in the logs:

in_interface=eth0 out_interface=eth1

Do you see it feasible?

Thanks!

[vbohata]

I noticed there is now observer.egress/ingress for firewall like devices. But seems still no interface field for regular devices like servers.

[willemdh]

Jumping into the ip range thing.

Log: 1.24.17.19 was in range 10.0.0.0-10.255.255.255

Grok: %{IP:client.ip} was in the range %{NOTSPACE:network.cidr}

10.0.0.0-10.255.255.255 = 10.0.0.0/8

Is there any place to put this yet? network.cidr seems appropriate. Anyone knows a way to convert 10.0.0.0-10.255.255.255 to 10.0.0.0/8 in Logstash?

[zez3]

Good point, should we make the field an ip_range then?

Also nice to see that ip addresses can be queried with a cidr notation: https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/range.html#ip-range

[FlorianHeigl]

I just stumbled over this when testing Elastic Defend...

i.e. with a rule as this one:

OS IS Mac AND process.name IS netbiosd AND process.code_signature.status IS trusted AND destination.port IS ONE OF [ 139 445 ]

I would like to filter out access to the local subnet, or at least allow certain subnets to make sure I'm tracking strange things and not regular use there audit logs or the agent on the server need to do the heavy lifting.

maybe this will already work, I don't know enough about it. but it seems that the issue is still open.