search

jasonish / evebox

Web Based Event Viewer (GUI) for Suricata EVE Events in Elastic Search
https://evebox.org/
MIT License
432 stars 67 forks source link

Data in elasticsearch not visible in EveBox #96

Closed thehilll closed 5 years ago

thehilll commented 5 years ago

I have just installed

EveBox 0.10.1 ES 6.5.4 Kibana 6.5.4

On a single host and then Filebeat 6.5.4 and Suricata 4.1.2 on a separate host. Events are being shipped to Elasticsearch using Filebeat's Suricata module with very basic settings:

output.elasticsearch:
  protocol: https
  hosts: ["host.domain:443"]
  path: /es/
  username: "username"
  password: "${ES_PWD}"

When I view the Kibana templates that this module installs I see events. However, when I view EveBox all panels are empty (including an All Events search). If I run it in verbose mode things seem fairly normal:

# evebox server -c /etc/evebox/evebox.yaml --verbose
2019-01-17 09:22:37 (server.go:178) <Info> -- This is EveBox Server version 0.10.1 (rev: c17f18b); os=linux, arch=amd64
2019-01-17 09:22:37 (server.go:267) <Info> -- Self test: found embedded index.html.
2019-01-17 09:22:37 (geoip-service.go:44) <Warning> -- Failed to initialize geoip database: no database files found
2019-01-17 09:22:37 (configdb.go:52) <Info> -- Using in-memory configuration DB.
2019-01-17 09:22:37 (migrator.go:68) <Debug> -- Initializing database.
2019-01-17 09:22:37 (migrator.go:79) <Info> -- Updating database to version 0.
2019-01-17 09:22:37 (migrator.go:79) <Info> -- Updating database to version 1.
2019-01-17 09:22:37 (server.go:320) <Info> -- Configuring ElasticSearch datastore
2019-01-17 09:22:37 (server.go:321) <Info> -- Using ElasticSearch URL http://127.0.0.1:9200
2019-01-17 09:22:37 (server.go:323) <Info> -- Using ElasticSearch Index filebeat.
2019-01-17 09:22:37 (elasticsearch.go:111) <Info> -- Event base index: filebeat
2019-01-17 09:22:37 (elasticsearch.go:112) <Info> -- Event search index: filebeat-*
2019-01-17 09:22:37 (server.go:353) <Info> -- Connected to Elastic Search (version: 6.5.4)
2019-01-17 09:22:37 (server.go:131) <Info> -- Session reaper started
2019-01-17 09:22:37 (server.go:165) <Info> -- Authentication disabled.
2019-01-17 09:22:37 (server.go:222) <Debug> -- Apply reverse proxy handler
2019-01-17 09:22:37 (server.go:276) <Info> -- Listening on 0.0.0.0:5636
2019-01-17 09:22:41 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from x.x.x.x
2019-01-17 09:22:41 (datastore-alertquery.go:155) <Info> -- Query elapsed time: 257.210243ms
2019-01-17 09:23:18 (anonymous.go:64) <Info> -- Logging in anonymous user {anonymous} from x.x.x.x
2019-01-17 09:23:37 (server.go:127) <Debug> -- Reaping sessions.
2019-01-17 09:24:37 (server.go:127) <Debug> -- Reaping sessions.
2019-01-17 09:25:37 (server.go:127) <Debug> -- Reaping sessions.

The only thing a little odd is Failed to initialize geoip database: no database files found as I do have the path properly set within /etc/evebox/evebox.yaml

geoip:
  disabled: false
  # Path to the MaxMind database. This must be the version 2 database
  # (http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz)
  # File must be ungzipped.
  #
  # This is temporary, EveBox will eventually support downloading and
  # updateing the geo database itself.
  database: /path/to/GeoLite2-City.mmdb

Where the actual path does exist. Even so, that doesn't seem likely to prevent anything from showing up.

The search index looks right. Is there anything else I can do to uncover where my error is?

Thanks a lot.

jasonish commented 5 years ago

The issue here is that the Filebeat Suricata module transforms the Suricata EVE events into the Elastic Common Schema (ECS) format and Evebox doesn't know how to work with events in that format yet.

I've created a ticket to support ECS here - https://github.com/jasonish/evebox/issues/97 but don't have an estimated completion time yet.

The workaround is to not use the Suricata module in Filebeat but instead send the Suricata eve log to Elastic Search in a more traditional way. I believe Filebeat can do this directly without any modules. Or you could use elastic search, or you could look at using EveBox in agent mode, or the evebox esimport tool.

thehilll commented 5 years ago

OK, thank you very much for this. We are new to this, and I had noticed that filebeat to logstash is much more common. I went with the module because it seemed like the easiest path in the future, but maybe it is a little new for other tools to be integrated.

Filebeat can send straight to logstash which then appears fine in the current version of EveBox. For anyone doing this with the 6.5 ELK stack and Ubuntu 18, this is a good tutorial:

https://www.howtoforge.com/tutorial/suricata-with-elk-and-web-front-ends-on-ubuntu-bionic-beaver-1804-lts/

There have been a number of changes to ELK that make older logstash filters break apparently.

jasonish commented 5 years ago

Closing. #97 will track ECS support.