ELK 6.3.0

[rndrev]

I am trying to integrate SweetSecurity sensor install on a Raspberry Pi and the Wazuh ELK 6.2.4 stack. Unfortunately it appears that there was a major change going from ELK 5.x.x to ELK 6.x.x, which is detailed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

More detailed information is here: https://logz.io/blog/removal-elasticsearch-mapping-types/

Does anyone know how I can update the install files so that they would work with ELK 6.2.4?

[rndrev]

I thought I would give some extra comments. I was able to get logstash to send the logs to the ELK server. The main issues is the mapping types cannot be imported, which messes up the processing of the log files.

Also, before I get attacked by the zealots...I know that ELK 5.5.1 is currently the latest version for the Raspberry Pi, so I am asking for it to work with a version that isn't used by the Pi. But it would be nice to have a version where you could send logs to a modern ELK server.

I will of course keep trying to get it working, and maybe I can learn enough to do that!

[rndrev]

Well, they have recently released ELK 6.3.0, so I think that is the target. I was able to get it working, but I had to install SweetSecurity first, and then rebuild the ELK stack up to 6.3.0. This took lots of time, but it appears to have worked. It would be easier if it would simply install directly though!

[ghost]

I'm working to do the same thing, I managed to get as far as installing 6.3.0 elasticsearch no problem. I installed kibana as well, but when it tries to install the kibana dashboards it fails with the following error:

Narrowed it down to json files in /kibana/dashboards, specifically the baselines.json. It looks like it is simply defining the dashboards and visualizations in this section.

I'm thinking it should be simple to convert these to single mapping, but I honestly don't know enough about elasticsearch mapping to understand how. I'll keep looking at the documentation and see if I can figure it out, although I'm not sure converting these dashboard will be the final solution to upgrading the ELK stack in sweet security to 6.3.0.

[rndrev]

@blakesplay: Here is the post I put on the ELK forums. It gives example error messages I was getting when trying to compile.

ELK 6.3.0 is way better than previous versions because they bundled it with the X-Pack.

I guess for me, maybe this issue is closed. I will leave this thread open though in case Travis needs it to assist with troubleshooting when he upgrades for a never version of ELK.

[ghost]

@rndrev Yeah I ended up upgrading the internal python files to use the last version of ELK 5 (5.6.10), installed x-pack, and used upgrade assistant to re-index kibana. Then it was a simple upgrade to 6.0.1. I'm planning on making an upgrade script to make this process a little easier on others.

I agree ELK 6.0 was a seriously good upgrade, I just wish they had an easier to access tool to allow re-indexing the definition of a set.

[rndrev]

@blakesplay: Yes, another thing I will mention is to make sure you do a client/server install. I found that having the client separate really cuts down on the amount of issues, because in the worst case (happens to me several times already) you simply can re-install the client from scratch.

This has shown me the importance of having more developers on a project. The Wazuh project updated their software in a matter of weeks to be more compliant. I realize that Travis is only one developer though, and this is a side project for him as far as I can tell.

Anyway, my goal was to use Bro for network monitoring so installing SweetSecurity was an easier way to do that than repackaging it all myself.

I should point out that my implementation also uses Wazuh 3.3 which means that the sequence of install should be:

1. Install SweetSecurity Server on Ubuntu OS.
2. Do the upgrade and install of latest version of ELK that Wazuh will support.
3. Upgrade SweetSecurity information by re-indexing.
4. Install Wazuh latest version plugins, etc.
5. Restore the default logstash template. SweetSecurity messes with this a bit, and when I was trying to get the logs from the PFSense sent to the ELK stack this was causing problems.
6. Do the SweetSecurity Client install on the Raspberry Pi 3 B+ and get it to report back to the server.

Anyway, so far I have integrated SweetSecurity (Bro), Wauzh, and PFSense together. Still working on getting the SNORT logs over (SNORT is running on the PFSense firewall as well).

Note: There may be major issues when they release ELK 7.0 as it is going to NOT allow the mappings SweetSecurity uses at all. Right now, the only reason this install process will work is because ELK keeps backwards compatibility for version 6 ONLY.

[marcellos]

I've been playing with migrating the installation scripts from Python to Ansible. For now it is still a one-to-one draft just for learning. Maybe interesting to someone in the future to make it easier to maintain/upgrade etc...

[ghost]

@rndrev Nice developments, seems like you have this thing figured out. You may want to consider making a raspberry pi image once you get things stable, rather than going through the whole process all over again. (that bro compilation seems to take ages!)

Anyway, I managed to create that upgrade script, still in the process of testing it. I found something called elasticdump, which allows me to grab specific indices and put them into json. Instead of dealing with installing x-pack, I just grab the indices from my working server and do a straight upgrade to 6.3.1. I then can upload the indices and everything works perfectly.

Good find with the Wazuh, looks interesting I'll have to give it a look.

The ELK 7.0 issue is going to be a serious problem. I wish I knew enough about elastic search indices to upgrade them. It's not an immediate issue though.