search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 595 forks source link

Fielddata is disabled on text fields by default. Set fielddata=true on [flow.client_hostname] #120

Closed johntdyer closed 6 years ago

johntdyer commented 6 years ago

Version Elasticflow 3.0.3 Logstash 6.2.1 Kibana 6.2.1

Error:


Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [flow.client_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a …

more
Less Info
OK
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.client_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elastiflow-3.0.3-2018.05.31","node":"HMp0-aWSRoCc2-tIvtNi7Q","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.client_hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}]},"status":400}
    at https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:580947
    at Function.Promise.try (https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:479242)
    at https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:478630
    at Array.map (<anonymous>)
    at Function.Promise.map (https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:478588)
    at callResponseHandlers (https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:580525)
    at https://xxxxxxx.server.com/kibana/bundles/commons.bundle.js?v=16627:21:569871
    at processQueue (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:132456)
    at https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:133349
    at Scope.$digest (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:144239)
    at Scope.$apply (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:147018)
    at done (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:100026)
    at completeRequest (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:104697)
    at XMLHttpRequest.xhr.onload (https://xxxxxxx.server.com/kibana/bundles/vendors.bundle.js?v=16627:116:105435)

Any idea whats going on here??

robcowart commented 6 years ago

It looks like the index template wasn't loaded. The default path was wrong in the Elasticsearch output, so if you didn't set it correctly in the environment variable it wouldn't have loaded.

The default path fix has been committed already for the next release. For now you can either edit the output directly or set the template path environment variable and reload Logstash. Templates are only applied to newly created indices so you will need to remove the ones written without the template.

robcowart commented 6 years ago

@johntdyer were you able to get this working?

moodymob commented 6 years ago

I've got the same issue:

Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

less Less Info OK Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elastiflow-2018.06.12","node":"acvUzeZxQUK_x87t-0BlmA","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}]},"status":400}

here is the checksum of the template file : 7a33f389672ec77770116ea4a53b9815 elastiflow.template.json

Here is the trace in my log:

[2018-06-12T14:10:43,382][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"} [2018-06-12T14:10:43,456][INFO ][logstash.outputs.elasticsearch] {"path_match"=>"netflow.xlate_dst_addr_ipv6", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_dst_port"=>{"path_match"=>"netflow .xlate_dst_port", "mapping"=>{"type"=>"long"}}}, {"netflow.xlate_src_addr_ipv4"=>{"path_match"=>"netflow.xlate_src_addr_ipv4", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_addr_ipv6"=>{" path_match"=>"netflow.xlate_src_addr_ipv6", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_port"=>{"path_match"=>"netflow.xlate_src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.agent_ip"=

{"path_match"=>"sflow.agent_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.drops"=>{"path_match"=>"sflow.drops", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_ip"=>{"path_match"=>"sflow.dst_ip", "m apping"=>{"type"=>"ip"}}}, {"sflow.dst_mac"=>{"path_match"=>"sflow.dst_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.dst_mask_len"=>{"path_match"=>"sflow.dst_mask_len", "mapping"=>{"type"= "long"}}}, {"sflow.dst_port"=>{"path_match"=>"sflow.dst_port", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_priority"=>{"path_match"=>"sflow.dst_priority", "mapping"=>{"type"=>"long"}}}, {"sf low.dst_vlan"=>{"path_match"=>"sflow.dst_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.eth_dst"=>{"path_match"=>"sflow.eth_dst", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_src"=>{"path_ma tch"=>"sflow.eth_src", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_type"=>{"path_match"=>"sflow.eth_type", "mapping"=>{"type"=>"long"}}}, {"sflow.flow_sequence_number"=>{"path_match"=>"sfl ow.flow_sequence_number", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length"=>{"path_match"=>"sflow.frame_length", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length_times_sampling_rate"= {"path_match"=>"sflow.frame_length_times_sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.header_size"=>{"path_match"=>"sflow.header_size", "mapping"=>{"type"=>"long"}}}, {"sflow.inpu t_interface"=>{"path_match"=>"sflow.input_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_format"=>{"path_match"=>"sflow.input_interface_format", "mapping"=>{"type"=>"long "}}}, {"sflow.input_interface_value"=>{"path_match"=>"sflow.input_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_address_next_hop_router"=>{"path_match"=>"sflow.ip_addressnext hop_router", "mapping"=>{"type"=>"ip"}}}, {"sflow.ip_checksum"=>{"path_match"=>"sflow.ip_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_dscp"=>{"path_match"=>"sflow.ip_dscp", "mapping" =>{"type"=>"long"}}}, {"sflow.ip_ecn"=>{"path_match"=>"sflow.ip_ecn", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_flags"=>{"path_match"=>"sflow.ip_flags", "mapping"=>{"type"=>"long"}}}, {"sflo w.ip_fragment_offset"=>{"path_match"=>"sflow.ip_fragment_offset", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_header_length"=>{"path_match"=>"sflow.ip_header_length", "mapping"=>{"type"=>"long "}}}, {"sflow.ip_identification"=>{"path_match"=>"sflow.ip_identification", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_next_header"=>{"path_match"=>"sflow.ip_next_header", "mapping"=>{"type"= "long"}}}, {"sflow.ip_packet_length"=>{"path_match"=>"sflow.ip_packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_priority"=>{"path_match"=>"sflow.ip_priority", "mapping"=>{"type"=>" long"}}}, {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_total_length"=>{"path_match"=>"sflow.ip_total_length", "mapping"=>{"type"=>"long "}}}, {"sflow.ip_ttl"=>{"path_match"=>"sflow.ip_ttl", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_type"=>{"path_match"=>"sflow.ip_type", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_version"=>{"p ath_match"=>"sflow.ip_version", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface"=>{"path_match"=>"sflow.output_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_for mat"=>{"path_match"=>"sflow.output_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_value"=>{"path_match"=>"sflow.output_interface_value", "mapping"=>{"type"=>"long "}}}, {"sflow.packet_length"=>{"path_match"=>"sflow.packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.padded"=>{"path_match"=>"sflow.padded", "mapping"=>{"type"=>"long"}}}, {"sflow.prot ocol"=>{"path_match"=>"sflow.protocol", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sample_length"=>{"path_match"=>"sflow.sample_length", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_pool"=>{ "path_match"=>"sflow.sample_pool", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_seq_number"=>{"path_match"=>"sflow.sample_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sampling_rate"= {"path_match"=>"sflow.sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.sflow_type"=>{"path_match"=>"sflow.sflow_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sflow_version"=>{"path _match"=>"sflow.sflow_version", "mapping"=>{"type"=>"long"}}}, {"sflow.size_header"=>{"path_match"=>"sflow.size_header", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index"=>{"path_match "=>"sflow.source_id_index", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index_name"=>{"path_match"=>"sflow.source_id_index_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.source_id_typ e"=>{"path_match"=>"sflow.source_id_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_ip"=>{"path_match"=>"sflow.src_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.src_mac"=>{"path_match"=>"sf low.src_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_mask_len"=>{"path_match"=>"sflow.src_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.src_port"=>{"path_match"=>"sflow.src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.src_priority"=>{"path_match"=>"sflow.src_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.src_vlan"=>{"path_match"=>"sflow.src_vlan", "mapping"=>{"ty pe"=>"long"}}}, {"sflow.stripped"=>{"path_match"=>"sflow.stripped", "mapping"=>{"type"=>"long"}}}, {"sflow.sub_agent_id"=>{"path_match"=>"sflow.sub_agent_id", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_ack_number"=>{"path_match"=>"sflow.tcp_ack_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_checksum"=>{"path_match"=>"sflow.tcp_checksum", "mapping"=>{"type"=>"long"}}}, {"sf low.tcp_flags"=>{"path_match"=>"sflow.tcp_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_header_length"=>{"path_match"=>"sflow.tcp_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow. tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seqnumber", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {" sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime _in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match "=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan _type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "eve nt"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=> {"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "city"=>{"type"=>"keyword"}, "c lient_addr"=>{"type"=>"ip"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "clie nt_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "country"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_addr"=>{"type"=>"ip"}, "dst_addr_trans "=>{"type"=>"ip"}, "dst_asn"=>{"type"=>"long"}, "dst_autonomous_system"=>{"type"=>"keyword"}, "dst_city"=>{"type"=>"keyword"}, "dst_country"=>{"type"=>"keyword"}, "dst_geo_location"=>{"type" =>"geo_point"}, "dst_hostname"=>{"type"=>"keyword"}, "dst_mac"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port"=>{"type"=>"long"}, "dst_porttrans"=>{"type"=>"long"}, "dst port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "ip_protocol"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "next_hop"=>{" type"=>"ip"}, "output_snmp"=>{"type"=>"keyword"}, "packets"=>{"type"=>"long"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "serveraddr"=>{"type"=>"ip"}, "server asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point "}, "server_hostname"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{" type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=> {"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{" type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}, "aliases"=>{}}} [2018-06-12T14:10:43,525][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.0.3

robcowart commented 6 years ago

Did you ensure that the index template was loaded as I mentioned above?

moodymob commented 6 years ago

I'm sure, i'm running a full ELK Stack 6.2.4

moodymob commented 6 years ago

Conf 30_output.logstash.conf: output { elasticsearch { hosts => [ "XX.XX.XX.XX:9200", "XX.XX.XX.XX:9200", "XX.XX.XX.XX:9200" ] index => "elastiflow-%{+YYYY.MM.dd}" template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/elastiflow/templates}/elastiflow.template.json" template_name => "elastiflow-3.0.3" template_overwrite => "true" } }

moodymob commented 6 years ago

My bad ! i just found the issue, i didn't put the right index name.

Now it work's

robcowart commented 6 years ago

Yeah. Without the version portion in the index name it will not match the template.

Versioning was added in 3.x so that in the future if there are schema changes it will be easier to reindex old data to the new schema.

vpiserchia commented 6 years ago

I see something weird here in the mappings: {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}} while after you have this: "ip_protocol"=>{"type"=>"keyword"},

is that expected and wanted?

regards

robcowart commented 6 years ago

That is fine. The original sflow data is a number (6, 17, etc). The normalized form is a string (TCP, UDP, etc). You shouldn't even see the sflow.ip_protocol field show up unless something weird happens. It is a dynamic mapping so the field won't be seen in Kibana until it shows up in the indexed data the first time.

VanDuy91 commented 5 years ago

Hello, it's work with a Elasticsearch node but with 3 nodes is not. ELK version: 6.1.3 Elastiflow version: 2.1.0

30_output.logstash.conf output { elasticsearch { hosts => [ "192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200" ]

user => "${ELASTIFLOW_ES_USER:elastic}"

#password => "${ELASTIFLOW_ES_PASSWD:changeme}"
index => "elastiflow-%{+YYYY.MM.dd}"
template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/templates}/elastiflow.template.json"
template_name => "elastiflow"
template_overwrite => "true"

} }

192.168.1.1-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: true node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]

discovery.zen.minimum_master_nodes: 3

indices.queries.cache.size: 30%

192.168.1.2-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: false node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]

discovery.zen.minimum_master_nodes: 3

indices.queries.cache.size: 30%

192.168.1.3-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: false node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]

discovery.zen.minimum_master_nodes: 3

indices.queries.cache.size: 30%

Has data in the system image

robcowart commented 5 years ago

If you look at a document in the Discover app, do you see fields whose name ends with .keyword? As this thread discusses it is likely that there was an issue loading the template.

It seems that you are just starting out, so I recommend that you use the latest release.

robcowart commented 5 years ago

BTW, you have an extra quote in all of these lines: discovery.zen.ping.unicast.hosts: [""192.1, and this "129.168.1 should probably be 192, not 129.

VanDuy91 commented 5 years ago

BTW, you have an extra quote in all of these lines: discovery.zen.ping.unicast.hosts: [""192.1

I just remove an quote but it still not work

robcowart commented 5 years ago

A few other things: Why do you have only one master node? If that node is down, the whole cluster will be down.

indices.queries.cache.size: 30% This is unlikely to help you much. Since the queries for the visualizations within the dashboards are typically loaded using a time period such as now-1hr the time, and thus the query, will be different each time the dashboard is loaded or refreshed. Since the queries are different they won't use any results from the query cache.

VanDuy91 commented 5 years ago

If you look at a document in the Discover app, do you see fields whose name ends with .keyword? As this thread discusses it is likely that there was an issue loading the template.

It seems that you are just starting out, so I recommend that you use the latest release.

Just Elastiflow 2.1.0 support ELK 6.1.3 I don't see anything end with .keyword. With three master nodes: curl http://localhost:9200/_cluster/health?pretty { "cluster_name" : "elastic-cluster", "status" : "red", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 12, "active_shards" : 24, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 8, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 75.0

"Stutus: red" --> it may cause to this error?

VanDuy91 commented 5 years ago

Other things: I comment out: indices.queries.cache.size: 30% In Kibana.yml elasticsearch.url: "http://192.168.1.1:9200"

robcowart commented 5 years ago

That is not a document using the Discover app. You have a screen shoot from the Discover app above, look at one of the documents.

If you are just starting out you can use a later version of Elasticsearch and Kibana, and still use 6.1.3 of Logstash.

robcowart commented 5 years ago

Just because Kibana is pointing to only one ES node doesn't mean that you can't have multiple master nodes.

VanDuy91 commented 5 years ago

I installed ELK ver 6.4 for the first time (#172). But it's not work then I downgrade to 6.2.4 (#155) Finally, I downgrade ELK to 6.1.3. Yes, it's work. And this error occur when I configured ES cluster.

robcowart commented 5 years ago

It would be best if you didn't keep jumping around. Pick a version and stick with that. You also need to follow the instructions EXACTLY. Given the thing that you had wrong in your config above, I am concerned that you aren't paying enough attention to the details to get everything setup correctly. When you keep switching versions of Elastic Stack and ElastiFlow, it is nearly impossible to help you.

VanDuy91 commented 5 years ago

It would be best if you didn't keep jumping around. Pick a version and stick with that. You also need to follow the instructions EXACTLY. Given the thing that you had wrong in your config above, I am concerned that you aren't paying enough attention to the details to get everything setup correctly. When you keep switching versions of Elastic Stack and ElastiFlow, it is nearly impossible to help you.

Thank you, I'm going to check around once again. Having somethings wrong in ES cluster configuration, I do think so. So interesting... :D Thanks again.

umarizulkifli commented 5 years ago

ES: 6.5.1 KI : 6.5.1 LS : 6.1.3 I had loaded index pattern via cli

curl -X POST localhost:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/root/elastiflow/kibana/elastiflow.index_pattern.json

However when i try to load the dashboard this error occur on ES.

image

org.elasticsearch.transport.RemoteTransportException: [elasticsearch.vqbn.com][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [flow.service_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
        at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:317) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:310) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.5.1.jar:6.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
PornCoinVal commented 5 years ago

I had some issue. But fixed it. Removed index-patern and shards Restarted cluster and upload index patern & wait 5 min and all start works

Rich07082 commented 5 years ago

I have the same issue after a couple days with my 3rd store. Currently running multi store with about 115k products. Store 1 & 2 don't have any elasticsearch issues. My 3rd store will have the following issue.

[2019-07-16 11:09:31] main.ERROR: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento224_product_4_v5","node":"E-X4XrXDS5m_7tGTjOEIUg","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400} [] []

After a couple days. Restart elasticsearch service and reboot server and it works for another couple days. I've been chasing this issue for months and unable to find the root cause. Any held would be appciated.

Currently running 2.3.2 and this issue has been since upgrading to 2.3.x. I don't think it was an issue prior. elasticsearch 6.5 php 7.2 mysql 5.7 debian

Thanks Rich

seeni9589 commented 4 years ago

@Rich07082 Currently I'm facing the same issue. Do you have an idea about this?

lux4rd0 commented 4 years ago

I know this is closed - but getting this on a brand new 4.0 beta pull to just try things out...

DrRoffe commented 4 years ago

Hi All,

trying with elasticsearch 7.5.2 + elastiflow 3.5.3 and the issue is still here. note: this is a single node of ELK 7.5.2.

did everything by the book. but this annoyance is very frustrating. probably i am still collecting everything but i can't view anything. getting this when opening an overview dashboard:

    "shard": 0,
    "index": "elastiflow-3.5.3-2020.02.17",
    "node": "FHQm8C6OSXGhorSso6Up2w",
    "reason": {
      "type": "illegal_argument_exception",
      "reason": "Fielddata is disabled on text fields by default. Set

fielddata=true on [flow.service_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can, however, use significant memory. Alternatively use a keyword field instead." }

what is the procedure to overcome this? (means for an already installed instance)

Thank you Moshe.

On Thu, Jan 16, 2020 at 2:04 AM Dave Schmid notifications@github.com wrote:

I know this is closed - but getting this on a brand new 4.0 beta pull to just try things out...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/120?email_source=notifications&email_token=AF436FJK5W365A6H5YBQZ2TQ56P73A5CNFSM4FC7VJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJCIEWA#issuecomment-574915160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF436FIZ22MCDKUBS7HUBNLQ56P73ANCNFSM4FC7VJ5A .

-- Moshe Roffe, CTO RHCE - 803005755016943

robcowart commented 4 years ago

With Elastic Stack 7.5.x you need to use the ElastiFlow 4.0.0-beta release.

joeygeo commented 4 years ago

I have this issue with ES 6.8

robcowart commented 4 years ago

@coversine which version of ElastiFlow are you using?