Logstash Shard error

[akrog79]

Hello,

I received data and is ingested and procesed fine. But when new day starts, my ingestion give this error:

Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open

Any idea?

[enotspe]

mmmm, that is very weird. How come you got so many shards??? Are you using datastreams?? Can you give me more details on how your indexes are being generated???

[akrog79]

the FortinetDragon works as you say in this repository, but I have ingested a PaloAlto FW and Fortigate FW without datastream, the logstash output generates an index every day.

[enotspe]

I recommend you follow the indexing strategy we have in FortiDragon: ILM and datastream per "type". That way you wont make and index per day which is not optimal.

[enotspe]

By the way, we also provide a dashboard for palo alto logs 💪

[akrog79]

So what we should do? Remove the date on the index name in logstash conf.d file and control it with index templates? I don't know how made a data stream.

I read abaout PaloAltgo on github! when will you post it? I'm looking forward to it, fortidragon is amazing and I haven't configured it 100% yet.

[enotspe]

The dashboard for Palo alto is already uploaded to the repo

https://github.com/enotspe/fortinet-2-elasticsearch/blob/master/kibana/panw%20ELK%20850.ndjson

For using our index strategy (ILM and datastreams), you have to follow the instructions provided

https://github.com/enotspe/fortinet-2-elasticsearch#on-kibana

and also setup your output pipeline for using datastreams, like the one provided on

https://github.com/enotspe/fortinet-2-elasticsearch/blob/master/logstash/conf.d/syslog-fortinet-common_ecs-output.conf#L531

[akrog79]

but you don't have PaloAlto integration, no?

[enotspe]

No. I have used filebeat module. It is not perfect, but at least it parses the most important logs. The problem is that i don't have an infrastructure where to test it with logstash.