[Filebeat] Field aliases yield confusing Kibana Discovery results

[andrewkroh]

When browsing the Kibana Discover tab I noticed that suricata.eve.timestamp and suricata.eve.flow.start were present for events that were NOT from Suricata. It's somewhat misleading to have the suricata.* fields rendered when it's not a suricata event.

These fields are not in the _source. They are aliases pointing to @timestamp and event.start, respectively, which did happen to exist in the events that I was browsing.

• https://github.com/elastic/beats/blob/e4686d03bef1ad0c88686ef07f316804508b3c06/x-pack/filebeat/module/suricata/eve/_meta/fields.yml#L180-L182
• https://github.com/elastic/beats/blob/e4686d03bef1ad0c88686ef07f316804508b3c06/x-pack/filebeat/module/suricata/eve/_meta/fields.yml#L679-L681

History

These aliases were added (https://github.com/elastic/beats/pull/10377) in order to avoid duplicating data storage in an event while still maintaining the original suricata EVE JSON structure.

Possible Solutions

• Remove the aliases and duplicate the data within _source.
• Remove the aliases and remove the suricata.* fields that renamed to an ECS field. Users familiar with the original EVE format might be a little lost until they learn ECS.
• Ideas?

---

Version: Filebeat 7.0.0 / Kibana 6.6.0

[elasticmachine]

Pinging @elastic/secops

[webmat]

But this kind of usage of aliases is to help migration from old schemas to the new ECS-centric schema. Is this a problem across the board, for example with fields such as source.address?

[ruflin]

I tried this out with the apache Filebeat module and -E migration.enabled which means all aliases for nginx, apache etc. are loaded. Looking at the event, I don't see also the alias fields. It seems the above only applies to date values? I'm not sure what the logic here is on the Kibana side.

Here a screenshot of an event:

[ruflin]

@andrewkroh Just noticed one difference in my tests: I tested against KB 7.0.0 Snapshot. Thought worth mentioning in case it makes a difference.

[sophiec20]

Found in 7.0.0-SNAPSHOT { "build" : { "hash" : "a90fcf0", "date" : "2019-02-20T18:17:35.110454Z" },

Loaded vanilla nginx logs into a clean instance and see the following fields in Discover which I don't think should be there.

Possibly not a kibana problem as the following return hits despite there not being any suricata data.

GET filebeat-*/_search
{
    "query": {
        "exists" : { "field" : "suricata.eve.timestamp" }
    }
}

[willemdh]

I'm seeing this suricata.eve.timestamp field in several datasets that have nothing to do with suricata.. (also we are not using suricata anywhere) Glad I found this issue, is there any plan to get rid of it?

[Jimmy-Singh]

We are using filebeat 7.3.0 on our Kubernetes cluster and noticed the "suricata.eve.timestamp" field popping up on every event. Did a bit of digging up and found that the field is set on the filebeat template; under suricata properties; as an alias to field @timestamp (line 3899).

"timestamp": { "path": "@timestamp", "type": "alias" }

Rectified the problem by removing those particular lines of code in the template and rolled over the index to a new one.

[J7mbo]

@Jimmy-Singh How exactly did you do this? You had to export the template or something first? What steps / commands can I run to do the same that you did?

[Jimmy-Singh]

You will have to modify the filebeat template; you can do so by exporting -> modifying -> import via CURL or by using the GET/PUT template API via the kibana dev tool ( https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html ).

Make sure to rollover the index after that ( https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-rollover-index.html )

[willemdh]

Why is this still in the latest beats templates?

It's very confusing for some users to see this in every filebeat source..

[willemdh]

@andrewkroh Even in 7.8.1 this is still in the template? Is there any technical reason why removal of "suricata.eve.timestamp" takes so much time? The number of times I had to explain this...

[andrewkroh]

It hasn't been changed because it would be a breaking change to remove the field, but I think the bugfix justifies the breaking change and we should remove it.

[johanhammar]

We updated to 7.12 and suricate.eve.timestamp is gone but now we started to see lots of other suricata fields (and we don't use suricata)

i.e suricata.eve.http.http_method suricata.eve.http.http_user_agent suricata.eve.http.length suricata.eve.http.status

(and also other fields like traefik.access.user_agent.original which we also don't use)

Are those also related to to aliases?

Sorry for commenting on a closed issue, let me know it should open a new.

[andrewkroh]

Something must have changed in Kibana to give the same behavior we saw with date types to other types. I'm in favor of removing those aliases from the Suricata module.

[andrewkroh]

I've opened a PR to remove all of the Suricata alias fields. https://github.com/elastic/beats/pull/26627

[andrewkroh]

I found only one more non-migration alias (aka one that's present in all mappings) that points to an ECS fields. That was traefik.access.user_agent.device. I think I'll remove that too while I'm making breaking changes.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[johanhammar]

👍

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[willemdh]

🦖

[J7mbo]

👍

[devamanv]

@andrewkroh We have seen an issue wherein a user reported seeing these suricate.eve.* fields in the events. They are using Filebeat v7.17.10. I see that a PR was raised to fix it as seen in your comment. I see that we have a label of backport-7.15.0, assuming it was only backported to Filebeat v7.15. Can you please help me understand or clarify if this issue could be seen in 7.17.x. Could really help us with our next steps.

[andrewkroh]

The commit from the backport PR appears in releases >= v7.15.0 so the suricata aliases should not be present in index templates from those builds.

$ git tag --contains 16108a
v7.15.0
v7.15.1
v7.15.2
v7.16.0
v7.16.1
v7.16.2
v7.16.3
v7.17.0
v7.17.1
v7.17.10
v7.17.11
v7.17.12
v7.17.13
v7.17.14
v7.17.2
v7.17.3
v7.17.4
v7.17.5
v7.17.6
v7.17.7
v7.17.8
v7.17.9

You can see what aliases are present in the index template by exporting it:

./filebeat export template | gron | grep '"alias"'

I checked 7.17 and there are no suricata alias fields.

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)