Closed johntdyer closed 6 years ago
It looks like the index template wasn't loaded. The default path was wrong in the Elasticsearch output, so if you didn't set it correctly in the environment variable it wouldn't have loaded.
The default path fix has been committed already for the next release. For now you can either edit the output directly or set the template path environment variable and reload Logstash. Templates are only applied to newly created indices so you will need to remove the ones written without the template.
@johntdyer were you able to get this working?
I've got the same issue:
Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
less Less Info OK Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"elastiflow-2018.06.12","node":"acvUzeZxQUK_x87t-0BlmA","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [flow.ip_protocol] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}]},"status":400}
here is the checksum of the template file : 7a33f389672ec77770116ea4a53b9815 elastiflow.template.json
Here is the trace in my log:
[2018-06-12T14:10:43,382][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"} [2018-06-12T14:10:43,456][INFO ][logstash.outputs.elasticsearch] {"path_match"=>"netflow.xlate_dst_addr_ipv6", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_dst_port"=>{"path_match"=>"netflow .xlate_dst_port", "mapping"=>{"type"=>"long"}}}, {"netflow.xlate_src_addr_ipv4"=>{"path_match"=>"netflow.xlate_src_addr_ipv4", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_addr_ipv6"=>{" path_match"=>"netflow.xlate_src_addr_ipv6", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_port"=>{"path_match"=>"netflow.xlate_src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.agent_ip"=
{"path_match"=>"sflow.agent_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.drops"=>{"path_match"=>"sflow.drops", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_ip"=>{"path_match"=>"sflow.dst_ip", "m apping"=>{"type"=>"ip"}}}, {"sflow.dst_mac"=>{"path_match"=>"sflow.dst_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.dst_mask_len"=>{"path_match"=>"sflow.dst_mask_len", "mapping"=>{"type"= "long"}}}, {"sflow.dst_port"=>{"path_match"=>"sflow.dst_port", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_priority"=>{"path_match"=>"sflow.dst_priority", "mapping"=>{"type"=>"long"}}}, {"sf low.dst_vlan"=>{"path_match"=>"sflow.dst_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.eth_dst"=>{"path_match"=>"sflow.eth_dst", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_src"=>{"path_ma tch"=>"sflow.eth_src", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_type"=>{"path_match"=>"sflow.eth_type", "mapping"=>{"type"=>"long"}}}, {"sflow.flow_sequence_number"=>{"path_match"=>"sfl ow.flow_sequence_number", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length"=>{"path_match"=>"sflow.frame_length", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length_times_sampling_rate"= {"path_match"=>"sflow.frame_length_times_sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.header_size"=>{"path_match"=>"sflow.header_size", "mapping"=>{"type"=>"long"}}}, {"sflow.inpu t_interface"=>{"path_match"=>"sflow.input_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_format"=>{"path_match"=>"sflow.input_interface_format", "mapping"=>{"type"=>"long "}}}, {"sflow.input_interface_value"=>{"path_match"=>"sflow.input_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_address_next_hop_router"=>{"path_match"=>"sflow.ip_addressnext hop_router", "mapping"=>{"type"=>"ip"}}}, {"sflow.ip_checksum"=>{"path_match"=>"sflow.ip_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_dscp"=>{"path_match"=>"sflow.ip_dscp", "mapping" =>{"type"=>"long"}}}, {"sflow.ip_ecn"=>{"path_match"=>"sflow.ip_ecn", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_flags"=>{"path_match"=>"sflow.ip_flags", "mapping"=>{"type"=>"long"}}}, {"sflo w.ip_fragment_offset"=>{"path_match"=>"sflow.ip_fragment_offset", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_header_length"=>{"path_match"=>"sflow.ip_header_length", "mapping"=>{"type"=>"long "}}}, {"sflow.ip_identification"=>{"path_match"=>"sflow.ip_identification", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_next_header"=>{"path_match"=>"sflow.ip_next_header", "mapping"=>{"type"= "long"}}}, {"sflow.ip_packet_length"=>{"path_match"=>"sflow.ip_packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_priority"=>{"path_match"=>"sflow.ip_priority", "mapping"=>{"type"=>" long"}}}, {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_total_length"=>{"path_match"=>"sflow.ip_total_length", "mapping"=>{"type"=>"long "}}}, {"sflow.ip_ttl"=>{"path_match"=>"sflow.ip_ttl", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_type"=>{"path_match"=>"sflow.ip_type", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_version"=>{"p ath_match"=>"sflow.ip_version", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface"=>{"path_match"=>"sflow.output_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_for mat"=>{"path_match"=>"sflow.output_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_value"=>{"path_match"=>"sflow.output_interface_value", "mapping"=>{"type"=>"long "}}}, {"sflow.packet_length"=>{"path_match"=>"sflow.packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.padded"=>{"path_match"=>"sflow.padded", "mapping"=>{"type"=>"long"}}}, {"sflow.prot ocol"=>{"path_match"=>"sflow.protocol", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sample_length"=>{"path_match"=>"sflow.sample_length", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_pool"=>{ "path_match"=>"sflow.sample_pool", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_seq_number"=>{"path_match"=>"sflow.sample_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sampling_rate"= {"path_match"=>"sflow.sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.sflow_type"=>{"path_match"=>"sflow.sflow_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sflow_version"=>{"path _match"=>"sflow.sflow_version", "mapping"=>{"type"=>"long"}}}, {"sflow.size_header"=>{"path_match"=>"sflow.size_header", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index"=>{"path_match "=>"sflow.source_id_index", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index_name"=>{"path_match"=>"sflow.source_id_index_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.source_id_typ e"=>{"path_match"=>"sflow.source_id_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_ip"=>{"path_match"=>"sflow.src_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.src_mac"=>{"path_match"=>"sf low.src_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_mask_len"=>{"path_match"=>"sflow.src_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.src_port"=>{"path_match"=>"sflow.src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.src_priority"=>{"path_match"=>"sflow.src_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.src_vlan"=>{"path_match"=>"sflow.src_vlan", "mapping"=>{"ty pe"=>"long"}}}, {"sflow.stripped"=>{"path_match"=>"sflow.stripped", "mapping"=>{"type"=>"long"}}}, {"sflow.sub_agent_id"=>{"path_match"=>"sflow.sub_agent_id", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_ack_number"=>{"path_match"=>"sflow.tcp_ack_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_checksum"=>{"path_match"=>"sflow.tcp_checksum", "mapping"=>{"type"=>"long"}}}, {"sf low.tcp_flags"=>{"path_match"=>"sflow.tcp_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_header_length"=>{"path_match"=>"sflow.tcp_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow. tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seqnumber", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {" sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime _in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match "=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan _type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "eve nt"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=> {"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "city"=>{"type"=>"keyword"}, "c lient_addr"=>{"type"=>"ip"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "clie nt_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "country"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_addr"=>{"type"=>"ip"}, "dst_addr_trans "=>{"type"=>"ip"}, "dst_asn"=>{"type"=>"long"}, "dst_autonomous_system"=>{"type"=>"keyword"}, "dst_city"=>{"type"=>"keyword"}, "dst_country"=>{"type"=>"keyword"}, "dst_geo_location"=>{"type" =>"geo_point"}, "dst_hostname"=>{"type"=>"keyword"}, "dst_mac"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port"=>{"type"=>"long"}, "dst_porttrans"=>{"type"=>"long"}, "dst port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "ip_protocol"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "next_hop"=>{" type"=>"ip"}, "output_snmp"=>{"type"=>"keyword"}, "packets"=>{"type"=>"long"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "serveraddr"=>{"type"=>"ip"}, "server asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point "}, "server_hostname"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{" type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=> {"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{" type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}, "aliases"=>{}}} [2018-06-12T14:10:43,525][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.0.3
Did you ensure that the index template was loaded as I mentioned above?
I'm sure, i'm running a full ELK Stack 6.2.4
Conf 30_output.logstash.conf: output { elasticsearch { hosts => [ "XX.XX.XX.XX:9200", "XX.XX.XX.XX:9200", "XX.XX.XX.XX:9200" ] index => "elastiflow-%{+YYYY.MM.dd}" template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/elastiflow/templates}/elastiflow.template.json" template_name => "elastiflow-3.0.3" template_overwrite => "true" } }
My bad ! i just found the issue, i didn't put the right index name.
Now it work's
Yeah. Without the version portion in the index name it will not match the template.
Versioning was added in 3.x so that in the future if there are schema changes it will be easier to reindex old data to the new schema.
I see something weird here in the mappings: {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}} while after you have this: "ip_protocol"=>{"type"=>"keyword"},
is that expected and wanted?
regards
That is fine. The original sflow data is a number (6, 17, etc). The normalized form is a string (TCP, UDP, etc). You shouldn't even see the sflow.ip_protocol field show up unless something weird happens. It is a dynamic mapping so the field won't be seen in Kibana until it shows up in the indexed data the first time.
30_output.logstash.conf output { elasticsearch { hosts => [ "192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200" ]
#password => "${ELASTIFLOW_ES_PASSWD:changeme}"
index => "elastiflow-%{+YYYY.MM.dd}"
template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/templates}/elastiflow.template.json"
template_name => "elastiflow"
template_overwrite => "true"
192.168.1.1-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: true node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]
192.168.1.2-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: false node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]
192.168.1.3-elasticsearch.yml path.data: /var/log/elasticsearch path.logs: /var/log/elasticsearch cluster.name: elastic-cluster node.name: ${HOSTNAME} node.master: false node.data: true node.attr.box_type: hot network.host: 0.0.0.0 http.port: 9200 discovery.zen.ping_timeout: 3s discovery.zen.ping.unicast.hosts: [""192.168.1.1:9200", "129.168.1.2:9200", "192.168.1.3:9200"]
Has data in the system
If you look at a document in the Discover app, do you see fields whose name ends with .keyword
? As this thread discusses it is likely that there was an issue loading the template.
It seems that you are just starting out, so I recommend that you use the latest release.
BTW, you have an extra quote in all of these lines: discovery.zen.ping.unicast.hosts: [""192.1
, and this "129.168.1
should probably be 192, not 129.
BTW, you have an extra quote in all of these lines:
discovery.zen.ping.unicast.hosts: [""192.1
I just remove an quote but it still not work
A few other things: Why do you have only one master node? If that node is down, the whole cluster will be down.
indices.queries.cache.size: 30%
This is unlikely to help you much. Since the queries for the visualizations within the dashboards are typically loaded using a time period such as now-1hr
the time, and thus the query, will be different each time the dashboard is loaded or refreshed. Since the queries are different they won't use any results from the query cache.
If you look at a document in the Discover app, do you see fields whose name ends with
.keyword
? As this thread discusses it is likely that there was an issue loading the template.It seems that you are just starting out, so I recommend that you use the latest release.
Just Elastiflow 2.1.0 support ELK 6.1.3 I don't see anything end with .keyword. With three master nodes: curl http://localhost:9200/_cluster/health?pretty { "cluster_name" : "elastic-cluster", "status" : "red", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 12, "active_shards" : 24, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 8, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 75.0
"Stutus: red" --> it may cause to this error?
Other things: I comment out: indices.queries.cache.size: 30% In Kibana.yml elasticsearch.url: "http://192.168.1.1:9200"
That is not a document using the Discover app. You have a screen shoot from the Discover app above, look at one of the documents.
If you are just starting out you can use a later version of Elasticsearch and Kibana, and still use 6.1.3 of Logstash.
Just because Kibana is pointing to only one ES node doesn't mean that you can't have multiple master nodes.
I installed ELK ver 6.4 for the first time (#172). But it's not work then I downgrade to 6.2.4 (#155) Finally, I downgrade ELK to 6.1.3. Yes, it's work. And this error occur when I configured ES cluster.
It would be best if you didn't keep jumping around. Pick a version and stick with that. You also need to follow the instructions EXACTLY. Given the thing that you had wrong in your config above, I am concerned that you aren't paying enough attention to the details to get everything setup correctly. When you keep switching versions of Elastic Stack and ElastiFlow, it is nearly impossible to help you.
It would be best if you didn't keep jumping around. Pick a version and stick with that. You also need to follow the instructions EXACTLY. Given the thing that you had wrong in your config above, I am concerned that you aren't paying enough attention to the details to get everything setup correctly. When you keep switching versions of Elastic Stack and ElastiFlow, it is nearly impossible to help you.
Thank you, I'm going to check around once again. Having somethings wrong in ES cluster configuration, I do think so. So interesting... :D Thanks again.
ES: 6.5.1 KI : 6.5.1 LS : 6.1.3 I had loaded index pattern via cli
curl -X POST localhost:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/root/elastiflow/kibana/elastiflow.index_pattern.json
However when i try to load the dashboard this error occur on ES.
org.elasticsearch.transport.RemoteTransportException: [elasticsearch.vqbn.com][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [flow.service_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:317) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:310) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.5.1.jar:6.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
I had some issue. But fixed it. Removed index-patern and shards Restarted cluster and upload index patern & wait 5 min and all start works
I have the same issue after a couple days with my 3rd store. Currently running multi store with about 115k products. Store 1 & 2 don't have any elasticsearch issues. My 3rd store will have the following issue.
[2019-07-16 11:09:31] main.ERROR: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento224_product_4_v5","node":"E-X4XrXDS5m_7tGTjOEIUg","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [category_ids] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400} [] []
After a couple days. Restart elasticsearch service and reboot server and it works for another couple days. I've been chasing this issue for months and unable to find the root cause. Any held would be appciated.
Currently running 2.3.2 and this issue has been since upgrading to 2.3.x. I don't think it was an issue prior. elasticsearch 6.5 php 7.2 mysql 5.7 debian
Thanks Rich
@Rich07082 Currently I'm facing the same issue. Do you have an idea about this?
I know this is closed - but getting this on a brand new 4.0 beta pull to just try things out...
Hi All,
trying with elasticsearch 7.5.2 + elastiflow 3.5.3 and the issue is still here. note: this is a single node of ELK 7.5.2.
did everything by the book. but this annoyance is very frustrating. probably i am still collecting everything but i can't view anything. getting this when opening an overview dashboard:
"shard": 0,
"index": "elastiflow-3.5.3-2020.02.17",
"node": "FHQm8C6OSXGhorSso6Up2w",
"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set
fielddata=true on [flow.service_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can, however, use significant memory. Alternatively use a keyword field instead." }
what is the procedure to overcome this? (means for an already installed instance)
Thank you Moshe.
On Thu, Jan 16, 2020 at 2:04 AM Dave Schmid notifications@github.com wrote:
I know this is closed - but getting this on a brand new 4.0 beta pull to just try things out...
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/120?email_source=notifications&email_token=AF436FJK5W365A6H5YBQZ2TQ56P73A5CNFSM4FC7VJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJCIEWA#issuecomment-574915160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF436FIZ22MCDKUBS7HUBNLQ56P73ANCNFSM4FC7VJ5A .
-- Moshe Roffe, CTO RHCE - 803005755016943
With Elastic Stack 7.5.x you need to use the ElastiFlow 4.0.0-beta
release.
I have this issue with ES 6.8
@coversine which version of ElastiFlow are you using?
Version Elasticflow 3.0.3 Logstash 6.2.1 Kibana 6.2.1
Error:
Any idea whats going on here??