Add a Network Zone drop-down to the Observability Hosts page

[mbudge]

Elastic-Agent can be deployed on multiple cloud providers, on-prem networks and data-centres. These networks can span multiple countries/regions/zones. Different network zones are often managed by different IT teams.

A managed service provider may have 100's/1000's of servers running in a data centre in multiple locations. A company may also use multiple cloud providers which host infrastructure in different regions/zones. On-prem networks can span across the globe when there a multiple sites in different countries. Security analysts and IT want to find Elastic-Agents in different network zones without having to refer to asset tracking spreadsheets.

To do this efficiently users need a way to select Elastic-Agents in under their control in a single field on the Observability > Infrastructure > Hosts page.

The zone tagging should be done using this feature in the Fleet policy settings

https://github.com/elastic/kibana/issues/179915

I recommend adding a new field to ECS to store the zone and environment tags, then add drop-down menu's to the Observability Hosts page to make it easier for IT to select Elastic-Agents in network zones they manage.

https://github.com/elastic/ecs/issues/2306

Zone/environment tagging should be done to both logs/metrics data.

Where Elastic-Agent is collecting data from external/api based services, engineers should use the beats processes to overwrite the zone/environment tags. For example, an Elastic-Agent might be running in GCP, but the integration is collecting Azure logs data. The beats process would be used to overwrite event.zone:gcp with event.zone:azure. The zone tagging logic could also be built into the elastic managed integrations.

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[willemdh]

The zone fields in the panw data are in observer.egress/ingress.zone

It would be awesome if we could use this to also store internal network zones of the related host. We could enrich with the jdbc filter from our cmdb with Logstash.

[roshan-elastic]

Hi @mbudge - thanks for raising this.

If I can play this back, I see two core problems here:

1. The ability to easily search/filter down by the "zone" in the Hosts UI Examples:

• I need to filter down by the "zone" without necessarily knowing the field name for zone, the values of the zone or even that I can search/filter by zone

e.g. Maybe I just know the name of my zone "foobar". I could start typing "foo..." and it immediately suggests "foobar" to me and I can filter by it (and I can see the other values and count of hosts in each "zone" with their status)

2. The ability to easily declare "zone" in a consistent way

• You could declare this as a 'namespace' but then it's mixed up with other namespace values which may have nothing to do with zone
• You could do this via tags but this has the same limitation
• You could them via these via a processor as a new field (e.g. zone) or via labels (e.g. labels.zone)

For (2), I'd assume you already know this but you would like a specific ECS field for "zone"?

Some initial feedback:

(1) Is something we'd like to offer. It's not in our short-term roadmap but we'd like to solve this problem (2) is something I'd like to understand a bit better but it looks like you would be able to do this via processors but it sounds like you'd like to standardise this as a common use case for ECS (or perhaps make it easier to declare this via fleet)?

[mbudge]

Businesses networks spread across multiple on-prem, data centre and cloud locations. In larger organisations each network zone tends to be managed by a different IT team. The aim is to have 1 field to select the network zone and 1 field to select the environment. All logs/metrics from all integrations we are running are tagged.

Security analysts use the zone field when they are responding to a security alert. Without the zone tag, they don't know which IT team to respond to, as we have 1000's of hosts spread across multiple on-prem network zones, data centres and cloud providers and it's not always clear in the host naming convention which network zone the alert from.

Zone tagging allows IT to select hosts they maintain in their network zone. They don't want to see the other 3000 hosts managed by another teams. Sometimes there is cross over, for example the data centre IT team might have an Azure account and the on-prem UK IT team might have 2 Azure accounts.

Here's some examples of useful fields for zone tagging

Network zone

"A network zone is a configurable boundary that you can use to grant or restrict access to computers and devices in your organization based on the IP address that is requesting access. You can define a network zone by specifying one or more individual IP addresses, ranges of IP addresses, or geographic locations."

https://help.okta.com/en-us/content/topics/security/network/network-zones.htm#:~:text=A%20network%20zone%20is%20a,address%20that%20is%20requesting%20access.

The ECS field could be network.zone, source.zone, event.zone or (network.zone and source.network.*)

equinix oci equinix-azure uk-azure1 uk-azure2 uk us ca ch aws gcp ibm external/api

Network Environment

The ecs field could be network.environment, source.environment, event.environment or (network.environment and source.network.*)

prod non-prod dev test uat sit prod dmz

Any type of log can be tagged as it's done at the collection layer. We went with event.zone and event.environment as the logs aren't just from network activity. host security logs firewall / network logs av logs dns logs application logs

The environment field is useful for both Observability / Security as it makes it easier to implement increased controls on production hosts/applications. It comes easier to select hosts from an environment as the tagging is done through the Fleet policies, so users don't have to maintain lists of hosts in each environment in elastic.

Yes standardise these fields in ECS to encourage developers to use these fields in Kibana.

The new data tagging feature in Fleet policies is going to make it easier to implement network zone tagging. We already use the Fleet processors to add these tags, but if we standardise the ECS fields then it reduces the chance of us having to re-build different processes we've built on elastic is in future.

[mbudge]

network.zone and network.environment might be a better field to tag the source of both host/network events.

[mbudge]

Another contender is

network.environment network.zone network.geo.* so we add network.geo.country_name and network.geo.city_name

[willemdh]

Imho we also need to correlate with datasets which contain a source / ingress and a destination / egress zone. So does this mean w'll also need a related.zones ?

[roshan-elastic]

Thanks everyone for this...we're currently working on a new Inventory and tackling this problem.

We may reach out to discuss...

[roshan-elastic]

In fact @mbudge / @willemdh, if you're interested in discussing this further - would you be able to reach out to me and we could perhaps jump on a call?