robcowart
commented
3 years ago Take a look at KNOWN_ISSUES.md and make sure that the max payload size for Kibana, and any proxy that it may be behind, has been increased.
Closed ken-crozier closed 3 years ago
robcowart
commented
3 years ago Take a look at KNOWN_ISSUES.md and make sure that the max payload size for Kibana, and any proxy that it may be behind, has been increased.
ken-crozier
commented
3 years ago max payload is set and there is no proxy between me and the elk server ... Something went bump in the night and a lot of things had been changed on the system, like heap size in kibana - I wonder if something else went south - how can I check on logstash indices ?
_# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# The maximum payload size in bytes for incoming server requests.
server.maxPayloadBytes: 8388608
# The Kibana server's name. This is used for display purposes.
server.name: "ELK.croziers.org"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://10.0.1.11:9200"]
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"_So I rebuilt the server on Ubuntu 20.04 with a lower version of ELK kibana/stable,now 6.8.13 amd64 [installed] elasticsearch/stable,now 6.8.13 all [installed] logstash/stable,now 1:6.8.13-1 all [installed]
Logstash loads the codec's and starts the pipeline
maxpayload is set to 8388608 in Kibana.yml
there is NO proxy between me and the server ... and I still can't load the elastiflow.kibana.7.8.x.ndjson file that I downloaded in the zip file ... So I'm stuck again ...
what else can I check or provide
robcowart
commented
3 years ago Now I see the issue. First,you should only use a release of ElastiFlow, not Master. Second, you need a release that is compatible with version of the Elastic Stack that you are using. Check the compatibility chart in INSTALL.md to determine the correct version. The 7.8.x Kibana config will not work with 6.x. There are features in the later config that weren't introduced until later versions of 7.x.
ken-crozier
commented
3 years ago Thanks, I have now upgrade ES and KB to 7.10.2 and left logstash at 6.8 - and was able to load the file. However I'm now getting a couple of errors, one is bad data and the other is all shards failed .... I don't see any errors thrown in the logstash logs ... I saw another post that mentioned something not loading with the template not loading ... where would I see that, if that's the case ???? here's the logstash startup log ......
"[2021-02-07T01:14:38,350][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.13"}
[2021-02-07T01:16:22,274][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-02-07T01:16:22,966][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@127.0.0.1:9200/]}}
[2021-02-07T01:16:23,357][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/"}
[2021-02-07T01:16:23,435][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2021-02-07T01:16:23,440][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2021-02-07T01:16:23,485][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2021-02-07T01:16:23,496][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}
[2021-02-07T01:16:23,707][INFO ][logstash.outputs.elasticsearch] ns"=>{"path_match"=>"sflow.tcp_options", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime_in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match"=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan_type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "agent"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ephemeral_id"=>{"type"=>"keyword"}, "hostname"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}, "version"=>{"type"=>"keyword"}}}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "client"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "destination"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "ecs"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"version"=>{"type"=>"keyword"}}}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"category"=>{"type"=>"keyword"}, "dataset"=>{"type"=>"keyword"}, "duration"=>{"type"=>"long"}, "end"=>{"type"=>"date"}, "kind"=>{"type"=>"keyword"}, "module"=>{"type"=>"keyword"}, "severity"=>{"type"=>"long"}, "start"=>{"type"=>"date"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "client_rep_tags"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_mac_oui"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_ifname"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "next_hop"=>{"type"=>"ip"}, "output_ifname"=>{"type"=>"keyword"}, "output_snmp"=>{"type"=>"keyword"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_rep_tags"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_mac_oui"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}, "wifi_sta_mac"=>{"type"=>"keyword"}, "wifi_ssid"=>{"type"=>"keyword"}, "wifi_wtp_mac"=>{"type"=>"keyword"}}}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}}}, "host"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"architecture"=>{"type"=>"keyword"}, "hostname"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "ip"=>{"type"=>"ip"}, "os"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"family"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}, "platform"=>{"type"=>"keyword"}, "version"=>{"type"=>"keyword"}}}}}, "log"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"level"=>{"type"=>"keyword"}}}, "message"=>{"type"=>"text", "norms"=>false}, "network"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=>{"type"=>"keyword"}, "bytes"=>{"type"=>"long"}, "iana_number"=>{"type"=>"long"}, "packets"=>{"type"=>"long"}, "transport"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "observer"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "egress"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"interface"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"alias"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "vlan"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "hostname"=>{"type"=>"keyword"}, "ingress"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"interface"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"alias"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "vlan"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}}}, "server"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "source"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"address"=>{"type"=>"keyword"}, "as"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"number"=>{"type"=>"keyword"}, "organization"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"name"=>{"type"=>"keyword"}}}}}, "bytes"=>{"type"=>"long"}, "domain"=>{"type"=>"keyword"}, "geo"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"city_name"=>{"type"=>"keyword"}, "country_name"=>{"type"=>"keyword"}, "country_iso_code"=>{"type"=>"keyword"}, "location"=>{"type"=>"geo_point"}}}, "ip"=>{"type"=>"ip"}, "mac"=>{"type"=>"keyword"}, "nat"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ip"=>{"type"=>"ip"}, "port"=>{"type"=>"long"}}}, "packets"=>{"type"=>"long"}, "port"=>{"type"=>"long"}, "registered_domain"=>{"type"=>"keyword"}, "top_level_domain"=>{"type"=>"keyword"}, "user"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "email"=>{"type"=>"keyword"}, "full_name"=>{"type"=>"keyword"}, "group"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"domain"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}, "hash"=>{"type"=>"keyword"}, "id"=>{"type"=>"keyword"}, "name"=>{"type"=>"keyword"}}}}}, "tags"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}}}
[2021-02-07T01:16:23,872][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-4.0.1
[2021-02-07T01:16:25,110][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2021-02-07T01:16:25,137][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2021-02-07T01:16:32,985][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2021-02-07T01:16:32,986][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2021-02-07T01:16:41,169][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"elastiflow", :thread=>"#
ken-crozier
commented
3 years ago ok so after about 20 hours .... everything is working thanks
HI, not sure what happened to my running system, but it stopped working and in trying to get it back I can't load the dashboards into Kibana. I know this has been discussed many times, and I did download the zip file, extracted it and used the file from it - but it will not load and I get the usual message of 'sorry file could not be processed'...
Everything is running, and logstash is receiving flows ... Any hints
Thanks Ken