Can´t use Filebeat index with evebox.yaml

[ngms17]

Hi.

When I start evebox with the config file, I´m not getting any events. My config file is the following.

# EveBox Server configuration file.

# Path to the data directory. This directory holds data for EveBox
# such as the configuration/user/authentication database, and SQLite
# database files if the sqlite database is being used. It needs to be
# writable by the user EveBox is running as. If not set it will
# default to the current directory.
#data-directory: /var/lib/evebox

# Database configuration.
database:

  # Database type: elasticsearch, sqlite.
  type: elasticsearch

  elasticsearch:
    url: http://elasticip:9200
    # production-style index
    index: filebeat
    # demo-style index
    # index: test-suricata
    disable-certificate-check: true

    # The keyword to use for terms query. EveBox will do its best to
    # figure this out on its own, but if you need to override it, you
    # can do so here. The usual values are:
    #    raw     -> Logstash / Elastic Search < 5.
    #    keyword -> Logstash / Elastic Search >= 5.
    #    ""      -> Filebeat / Elastic Search >= 5.
    # Note that a quoted empty string is required to force an empty string.
    #keyword: ""

    username: itsnottherealusername
    password: itsnottherealpassword

    # For loading the EveBox template (Logstash compatible) into
    # Elasticsearch. It is recommended to turn this option on if only
    # using EveBox to add events to Elasticsearch. Leave disabled if
    # already using Logstash or Filebeat on the same index.
    #
    # Default: false
    #force-template: false

With the -i flag, filebeat index seems to be reachable but I can´t start evebox because I have a secured elasticsearch node.

evebox server -e http://10.11.1.21:9200 -i filebeat

[jasonish]

Are you using 0.13.1? With a minimal config like:

database:
  type: elasticsearch
  elasticsearch:
    url: http://10.16.1.10:9200
    index: filebeat

And starting evebox like:

evebox server -c evebox.yaml -v

I see:

2021-05-03 15:42:11  INFO evebox::server::main: Found Elasticsearch version 7.11.2 at http://10.16.1.10:9200
2021-05-03 15:42:11 DEBUG evebox::server::main: Elasticsearch base index: filebeat
2021-05-03 15:42:11 DEBUG evebox::server::main: Elasticsearch search index pattern: filebeat-*
2021-05-03 15:42:11 DEBUG evebox::server::main: Elasticsearch ECS mode: false

Note that you can use -i on the command line to override the index in the configuration file:

evebox server -c evebox.yaml -v -i custom-index

[ngms17]

Still not getting any event on EveBox.

ngsantos@suricata-elk01:~$ evebox server -c /etc/evebox/evebox.yaml -v --host 10.11.1.21
2021-05-04 10:46:47  INFO evebox::version: This is EveBox version 0.13.1 (rev: 0dbcb12); x86_64-unknown-linux-musl
2021-05-04 10:46:47 DEBUG evebox::server::main: Certificate checks disabled: true
2021-05-04 10:46:47  INFO evebox::server::main: Using temporary in-memory configuration database
2021-05-04 10:46:47  INFO evebox::sqlite::init: Found event database schema version -1
2021-05-04 10:46:47  INFO evebox::sqlite::init: Initializing SQLite database (configdb)
2021-05-04 10:46:47  INFO evebox::sqlite::init: Updating SQLite database to schema version 1 (configdb)
2021-05-04 10:46:47 DEBUG evebox::sqlite::init: Did not find resource file configdb/V2.sql, database migration done
2021-05-04 10:46:47  INFO evebox::server::main: Found Elasticsearch version 7.12.0 at http://10.11.1.21:9200
2021-05-04 10:46:47 DEBUG evebox::server::main: Elasticsearch base index: filebeat
2021-05-04 10:46:47 DEBUG evebox::server::main: Elasticsearch search index pattern: filebeat-*
2021-05-04 10:46:47 DEBUG evebox::server::main: Elasticsearch ECS mode: false
2021-05-04 10:46:47  INFO evebox::server::main: Starting server on 10.11.1.21:5636, tls=false
2021-05-04 10:47:10 DEBUG evebox::server::asset: Loading asset index.html
2021-05-04 10:47:10 DEBUG evebox::server::asset: Loading asset styles.ba96b53d8799dc78590f.css
2021-05-04 10:47:10 DEBUG evebox::server::asset: Loading asset runtime.52d411db9a7ccbeecbe6.js
2021-05-04 10:47:10 DEBUG evebox::server::asset: Loading asset polyfills.35a5ca1855eb057f016a.js
2021-05-04 10:47:10 DEBUG evebox::server::asset: Loading asset main.99942c064cac91c87d1a.js
2021-05-04 10:47:11  INFO evebox::server::main: Creating anonymous session for user from 10.21.1.8 with name <anonymous>
2021-05-04 10:47:11 DEBUG evebox::server::asset: Loading asset favicon.ico
2021-05-04 10:47:11 DEBUG evebox::server::asset: Loading asset fontawesome-webfont.20fd1704ea223900efa9.woff2
2021-05-04 10:47:11 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:17 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:18 DEBUG evebox::elastic::eventstore: Rewriting tag escalated to evebox.escalated
2021-05-04 10:47:23 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:28 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:33 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:38 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:43 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:49 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:54 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:47:59 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:04 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:10 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:16 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:21 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:26 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:31 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:36 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:41 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:46 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:51 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:48:56 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:49:01 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:49:07 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:49:12 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived
2021-05-04 10:49:18 DEBUG evebox::elastic::eventstore: Rewriting tag archived to evebox.archived

Elastic is local to this machine. Suricata is sending events to Elastic via Filebeat from a remote machine. Index pattern and index are all ok

[jasonish]

Do you use the Filebeat Suricata module?

[ngms17]

yes, i use the suricata module

[jasonish]

Try adding --ecs to the comnand line. Its experimental, but the Filebeat Suricata module uses a new schema that EveBox doesn't fully support yet.

[ngms17]

It worked! Perfect!

One last question for now. How can i make "systemctl start evebox" to iniciate evebox exactly like the "evebox server -e /etc/evebox/evebox.yaml -v --host 10.11.1.21 --ecs"?

[jasonish]

If on Debian, look at /etc/default/evebox...

EVEBOX_OPTS="-c /etc/evebox/evebox.yaml -v --host 10.11.1.21 --ecs"

should do it.

[ngms17]

Try adding --ecs to the comnand line. Its experimental, but the Filebeat Suricata module uses a new schema that EveBox doesn't fully support yet.

Do you plan on giving full support to suricata module?

Maybe because of that, my evebox implementation seems to be incomplete. For example, clicking the link of a given flow gives me this:

Also, no information about flows

I noticed that these links always provide me "No events found". So i can´t make any correlations.

[jasonish]

I hope to. Its a bit like adding support for a whole new database unfortunately.

[ngms17]

So if i use logstash instead of the filebeat suricata module, all should be ok?

Or maybe consuming the logs directly from suricata?

[jasonish]

Yes. Or even if you use Logstash without the Suricata module, but instead its more generic output you will be OK.

The Suricata module for Filebeat remaps Suricata events into ECS format/schema, which is quote different from the Suricata event schema.

[jasonish]

Issue for ECS: https://github.com/jasonish/evebox/issues/97