search

sm-biz / paloalto-elasticstack-viz

PANW Firewall Visualisations using Elastic Stack
90 stars 39 forks source link

No rollover config/help #4

Open ih2010 opened 6 years ago

ih2010 commented 6 years ago

I've enjoyed the contents of this project immensely - it set me up with some wicked dashboards for a NOC TV at the beginning of the month.

Unfortunately, there is no built-in rollover capabilities to the ELKstack product line that apply to this project - and you have not mentioned anything of the sort here. Understandable, but unfortunate. Because of this, I filled up the 80 gig drive I was using within 2-3 weeks :(

Do you have any recommendations/help for rolling over data collected when setup in the way this project has specified? I'm about to wipe everything clean and follow this same guide, but taking the information from this managing time-based indices effectively article into account so that this can be used for more than 2 weeks at a time before needing a reset.

Before I dive deep into that and start from scratch on this (again) I'm hoping to find a practical example on how to best set this up taking PA FWs into account. Any ideas?

Thank you!

sm-biz commented 6 years ago

Hi oh2010

Yep, good point. I actually have an all-new version of this project running in my dev environment at present, that includes;

I'm going to write this new version up and put it on here shortly, and it should answer your questions. In the short term, if you want to log into the Elastic Stack element you're looking for is Curator. By re-configuring Logstash to output a per-day index, you can then run Curator actions (scripted via cron or Windows Task Scheduler) to trim off old data.

Examples coming soon!

ih2010 commented 6 years ago

Ahhhhh, this makes so much sense. I saw Elasticsearch Curator 5.5 and figured it was an older version meant to run on ElasticSearch 5.5 rather than 6.x. This is my missing piece of time-based log management for sure, thank you very much! I'll keep my eyes peeled for your post, but in the meantime hopefully I'll be able to get Curator up and running myself :)

On Sun, Jul 1, 2018 at 4:11 AM, Sam Miller notifications@github.com wrote:

Hi oh2010

Yep, good point. I actually have an all-new version of this project running in my dev environment at present, that includes;

  • Automated rollover (traffic 7 days, threat/config/system 30 days)
  • Integrated Netflow (via Elastiflow)
  • Better index management for Elastic
  • Better pipeline management for Logstash
  • Better dashboards for Kibana

I'm going to write this new version up and put it on here shortly, and it should answer your questions. In the short term, if you want to log into the Elastic Stack element you're looking for is Curator. By re-configuring Logstash to output a per-day index, you can then run Curator actions (scripted via cron or Windows Task Scheduler) to trim off old data.

Examples coming soon!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sm-biz/paloalto-elasticstack-viz/issues/4#issuecomment-401591625, or mute the thread https://github.com/notifications/unsubscribe-auth/ADQQEwS955ODkibG7f-yFtKm2VX7YjtTks5uCIRMgaJpZM4U9oN_ .