Closed psuhaj closed 3 years ago
The -i filebeat
option should use the index pattern filebeat-*
when searching.
I'm not sure why EveBox would be creating the index, that should only be done when configuring EveBox to also ship logs to Elasticsearch which is not the default.
I ran it with this command: evebox server -e http://localhost:9200 --host 0.0.0.0 -i filebeat
, is it ok?
I am checking it now and you are right, it did not create new index, it could be some older index which remained. Bud it still does not read the data from filebeat indices. The indices are named like this:
Can you share your filebeat configuration? Is it using json.keys_under_root: true
?
Yes it is using. Here is my filebeat.yaml
.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/suricata/eve.json
json.keys_under_root: true
overwrite_keys: true
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
setup.dashboards.enabled: false
output.elasticsearch:
hosts: ["http://localhost:9200"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
One more question, is the option database.elasticsearch.index in evebox.yaml
the same setting as specifiing the option -i
? When I set the index name to filebeat in evebox.yaml
then after start it creates other index, see the screenshot below. I have the option input.enabled set to false
. I start filebeat before starting evebox, so the index is created before evebox is started.
Yes, -i
, --index
, and database.elasticsearch.index
are all the same. Command line takes precedence over the configuration file.
I'm curious, in your setup does Filebeat add events to an index that has the "-date" suffix on it? Or something just like "filebeat" with no suffix. The search pattern used is "
What versions are you using? Elastic moves at a breaking change pace thats hard to keep up with. My last set of comprehensive testing was with 7.10, but I haven't tested 7.11.. I am using 7.11.2 in my own install, but I'm using the EveBox method of adding events to Elastic which is closer to the legacy way of Logstash than Filebeat.
Filebeat creates index like that first in the screenshot above, so the format is filebeat-{version}-{date}-{num}
, where num is starting with 000001 and is incrementing when indexes are being rotated by elastic ILM. I am using the latest versions (7.11.2).
Ok, found an issue. If using the default configuration file, and the input section exists, but is enabled, it still gets loaded so EveBox will create an index and start adding alerts.
Quick fix would be to:
input
section from the configuration file.Then, while I don't think this is required, but still good to clean things up:
But I'll be fixing the code for this right away as well.
Thank you. As I see it is fixed now, I will give it a try. Is the 0.13 the new stable version?
Thank you. As I see it is fixed now, I will give it a try. Is the 0.13 the new stable version?
Yeah, quite a few little fixes built up in there and thought that this was the tipping point for a release.
Fixed in 0.13.0.
Hello. I wanted to ask if it is any option to read from specified elasticsearch index? I ship suricata logs to elasticsearch using filebeat (not using the filebeat module, but setting it up in
filebeat.yml
). If I specify the option-i filebeat
then Evebox won't read from the indices created by filebeat (format isfilebeat-7.11.2-YY.mm.dd
), but it will create a new index in formatfilebeat-YY.mm.dd
. Is there any solution for this? I want to rotate my indices hourly using elastic index lifecycle management, which is applied on indices created by filebeat. Thank you.