Exception on elasticsearch: Fielddata is disabled on text fields by default. Set fielddata=true

[develoxir]

Hi,

in the past year about, I've been happily running elasticsearch 5.6.8 using evebox 0.9.0 on a raspberry Pi3 passing through them events coming from a suricata 4.0.3 running on an Asus wireless router. Everything has been working fine, except that lately things became a bit too sluggish, so I decided to treat myself with a dedicated SoC box (NUC style).

With this chance I installed ubuntu server 18.04, elasticsearch 6.5.4, evebox 0.10.1 (rev: c17f18b), both using the standard apt, while keeping the same suricata source for events.

I've configured everything "inspiring" myself with what I did on the raspi, and I see the alerts coming into the system (through rsyslog), written in a suricata log file, properly read by evebox, stored into elasticsearch and shown in evebox's GUI. But ... I get tons of exceptions in elasticsearch as the following one, and almost no report works on evebox's GUI:

[2019-01-10T11:00:32,439][DEBUG][o.e.a.s.TransportSearchAction] [I7knmjC] [filebeat-2019.01.10][3], node[I7knmjClSlClRpFB8sd0Ig], [P], s[STARTED], a[id=VuZZoMG-Rjqk-5p3NruweQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[filebeat-*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"query":{"bool":{"filter":[{"exists":{"field":"event_type","boost":1.0}},{"term":{"event_type":{"value":"alert","boost":1.0}}},{"range":{"@timestamp":{"from":"2019-01-03T11:00:32.432775655+01:00","to":null,"include_lower":true,"include_upper":true,"boost":1.0}}}],"must_not":[{"term":{"tags":{"value":"archived","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"@timestamp":{"order":"desc"}}],"aggregations":{"signatures":{"terms":{"field":"alert.signature_id","size":10000,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"sources":{"terms":{"field":"src_ip","size":10000,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"destinations":{"terms":{"field":"dest_ip","size":10000,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"escalated":{"filter":{"term":{"tags":{"value":"escalated","boost":1.0}}}},"newest":{"top_hits":{"from":0,"size":1,"version":false,"explain":false,"sort":[{"@timestamp":{"order":"desc"}}]}},"oldest":{"top_hits":{"from":0,"size":1,"version":false,"explain":false,"sort":[{"@timestamp":{"order":"asc"}}]}}}}}}}}}}}]
org.elasticsearch.transport.RemoteTransportException: [I7knmjC][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [src_ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
    at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670) ~[elasticsearch-6.5.4.jar:6.5.4]
    at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.5.4.jar:6.5.4]
    at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-6.5.4.jar:6.5.4]
    at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-6.5.4.jar:6.5.4]

I see lots of these not only on src_ip fields, but several others as well.

Did I miss some step or configuration ? Or, is anything I can do about this ?

Thanks in advance Peppe

[jasonish]

Can you clarify how you are adding events to Elastic Search? Does rsyslog drop them into a file, then you are using the evebox server itself to pick up the log and add it to Elastic Search?

[develoxir]

Hi Jason,

thanks for looking at this in the first place. Exactly, you got it right.

[jasonish]

If you haven't restarted the evebox server since first start, please do. There is an issue on first run, when it installs the Elastic Search template that it doesn't pickup the correct index configuration.

Restart it anyways and look at the log lines that are output in the first few moments:

2019-01-10 15:27:04 (server.go:323) <Info> -- Using ElasticSearch Index logstash.
2019-01-10 15:27:04 (elasticsearch.go:111) <Info> -- Event base index: logstash
2019-01-10 15:27:04 (elasticsearch.go:112) <Info> -- Event search index: logstash-*
2019-01-10 15:27:04 (server.go:353) <Info> -- Connected to Elastic Search (version: 6.5.4)
2019-01-10 15:27:04 (elasticsearch.go:200) <Info> -- Found templates [logstash]
2019-01-10 15:27:04 (elasticsearch.go:239) <Info> -- Found Elastic Search keyword suffix to be: keyword

Its the keyword suffix I want to check.

Also, can you check that rsyslog isn't modifying the logs in anyway? The file evebox is reading from needs to be pretty identical to the log file that suricata outputs.

[develoxir]

Uhm, perhaps you hit the problem, because this is the start log for my evebox instance and I can't see what you're looking for:

2019-01-10 18:19:35 (server.go:178) <Info> -- This is EveBox Server version 0.10.1 (rev: c17f18b); os=linux, arch=amd64
2019-01-10 18:19:35 (server.go:267) <Info> -- Self test: found embedded index.html.
2019-01-10 18:19:35 (configdb.go:59) <Info> -- Using configuration database file /var/lib/evebox/config.sqlite
2019-01-10 18:19:35 (server.go:320) <Info> -- Configuring ElasticSearch datastore
2019-01-10 18:19:35 (server.go:321) <Info> -- Using ElasticSearch URL http://localhost:9200
2019-01-10 18:19:35 (server.go:323) <Info> -- Using ElasticSearch Index logstash.
2019-01-10 18:19:35 (elasticsearch.go:111) <Info> -- Event base index: logstash
2019-01-10 18:19:35 (elasticsearch.go:112) <Info> -- Event search index: logstash-*
2019-01-10 18:19:35 (server.go:353) <Info> -- Connected to Elastic Search (version: 6.5.4)
2019-01-10 18:19:35 (elasticsearch.go:200) <Info> -- Found templates [logstash]
2019-01-10 18:19:35 (server.go:464) <Info> -- Configuring internal eve log reader
2019-01-10 18:19:35 (template.go:81) <Info> -- Loading template template-es6x.json for index logstash
2019-01-10 18:19:36 (rulemap.go:108) <Info> -- Loaded 25481 rules
2019-01-10 18:19:36 (server.go:131) <Info> -- Session reaper started
2019-01-10 18:19:36 (bookmarker.go:71) <Info> -- Using bookmark file /var/log/asus-ac5300/asus-suricata.log.bookmark
2019-01-10 18:19:36 (bookmarker.go:159) <Info> -- Found valid bookmark, jumping to offset 100
2019-01-10 18:19:36 (server.go:165) <Info> -- Authentication disabled.
2019-01-10 18:19:36 (server.go:276) <Info> -- Listening on 0.0.0.0:5636

And this in turn let me spot this piece of configuration:

    # The keyword to use for terms query. EveBox will do its best to
    # figure this out on its own, but if you need to override it, you
    # can do so here. The usual values are:
    #    raw     -> Logstash / Elastic Search < 5.
    #    keyword -> Logstash / Elastic Search >= 5.
    #    ""      -> Filebeat / Elastic Search >= 5.
    # Note that a quoted empty string is required to force an empty string.
    keyword: ""

which I had left as it was while, now reading it, I should have set it to:

keyword: "keyword"

Doing it seems to have solved the problem.

Thanks a lot Jason

[jasonish]

Please try a latest dev build from https://evebox.org/files/development/ - it fixes one known issue here. Might help.

Thanks.

[develoxir]

I confirm, issue solved thanks to your hint to look for the keyword suffix. My fault for not spotting the proper, different configuration to be applied in the new environment.

Thanks a lot Jason