field data disabled by default

[ahusking]

Hi All,

I'm using the docker elk image set from Here and then installing Elastiflow on top of the image.

I'm then sending Netflow V5 data in from one of our network devices and creating the index/importing the 6.4.x dashboards but get the field data is disabled by default error for half a dozen fields resulting in no dashboards loading.

I can confirm that flows are going into Elasticsearch and that when I click the flows dashboard, the flows load but nothing else does.

Any help would be great.

Regards,

[robcowart]

Usually when this issue occurs it is that for some reason Elasticsearch didn’t have the index template when the index was created, resulting in some fields having incorrect mappings.

[ahusking]

Hrm,

I've tried the following:

1. Build docker env.
2. Import index template.
3. Import dashboard/templates.
4. enable net-flow

and

1. Build docker env.
2. enable netflow
3. Import index template.
4. Import dashboard/templates.

I'll rebuild the environment tomorrow and confirm that importing the index template then the dashboards before enabling Netflow.

Thanks

[robcowart]

I think you are confusing the difference between Index Template (an Elasticsearch thing that describes how to create the new index) and Index Pattern (a Kibana thing that explains fields to Kibana).

[ahusking]

Ok,

So i've done the following from scratch today to attempt to get it working.

build/start the docker containers (not retaining any data). Import the index pattern into Kibana. Import the saved objects/dashboards into Kibana enabled netflow V5 to the logstash container.

Failed to load the dashboard with the following error (there are 6 different fields that have the same error).

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elastiflow-3.3.0-2018.10.29","node":"Cvs109eARcqQzCYOTnnKgw","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.server_hostname] in order to load fielddata in memory by uninverting the inverted index.

Have i missed something?

[robcowart]

It seems you are focused on Kibana, but you should be focused on Logstash. It is Logstash that manages the Index Template sent to Elasticsearch.

I suggest that you first get ElastiFlow working using a native Linux install, by following the instructions in INSTALL.md. Once you understand how to do that, you can then move on to Docker.

[ahusking]

Resolved. Seems it was a kibana issue, uploading the data using the docker script (from the fork), worked a treat.

[robcowart]

Glad you worked it out. Dockerizing Elastic in a "pre-populated" way is actually a pain. I've done some work to provide a fully dockerized "turnkey" release of ElastiFlow, but there are still a couple of things to work out. Hopefully I will have it released within the next couple of weeks.