Making Elasticsearch green

[michelpy]

I wanted to make elasticsearch status green on single-node. Here's what I did :

leafpad /etc/elasticsearch/elasticsearch.yml Uncomment : index.number_of_shards: 1 index.number_of_replicas: 0

curl -XDELETE 'http://localhost:9200/_all'

Now it's green allright. The curl command was a little brutal; all I miss are the dashboards in the "Stamus" menu, I do I bring them back ? I will re-customize.

Also, a curl command that clears the unassigned shards without killing everything else would not be a bad idea :P

[pevma]

yes - that curl command ... :) The main reason why right out of the box the status of elasticsearch in Scirius is yellow is because there is only one node in the cluster. More info can be found here: https://github.com/StamusNetworks/SELKS/wiki/Tuning-SELKS#elasticsearch-tuning-considerations

With regards to reloading the dashboards in SELKS: https://github.com/StamusNetworks/SELKS/wiki/How-to-load-or-update-dashboards

[michelpy]

yes - that curl command ... :)

:P I liked it though, now my elasticsearch is green and I have only 2 shards. Sometimes, using a nuke is entertaining.

Seriously, what did I lose that won't re-create itself ?

The main reason why right out of the box the status of elasticsearch in Scirius is yellow is because there is only one node in the cluster.

I know, and this specific one will always be a one-node cluster. Which is why I did this: index.number_of_shards: 1 index.number_of_replicas: 0

What curl command would you recommend to clear the unassigned shards after that ? They don't clean by themselves.

https://github.com/StamusNetworks/SELKS/wiki/How-to-load-or-update-dashboards

Worked fine, thanks for the pointer.

What is the proper way to backup a dashboard that has been customized, and to restore it possibly on a clean load ?

[pevma]

Seriously, what did I lose that won't re-create itself ?

Just the dashboards (and all of the data :) )

What is the proper way to backup a dashboard that has been customized, and to restore it possibly on a clean load ?

Have a look in /etc/init.d/kibana-dashboards-stamus - you will see there the scripts and what they do.

[b-u-g-s]

Hi, new to SELK and just done a fresh SELK 2.0 install followed by apt-get update/upgrade. I can see a yellow elasticsearch status with 10 unassigned shards.

As I said I am new to this and have new idea what the "shards" are but I guess because it is yellow it isn't good! ;)

how do I fix this? should I use the above curl command? I saw another post below that seems to be a similar solution: http://stackoverflow.com/questions/19967472/elasticsearch-unassigned-shards-how-to-fix

By the way, I can see alerts, traffic etc all ok in Kibana so something is working!

Thanks!

[michelpy]

Hi bugsland,

This is my pathetic understanding : Short version : if you are installing SELKS 2.0 from the default DVD install, elasticsearch will come out yellow. I don't think there is anything do do about it short of burning a custom .iso.

The core reason of elasticsearch being yellow is because it installs as a 2node-or-better cluster with only one node out of the minimum default of two. Right after the first reboot, it will be yellow.

Of course, you don't have another node, otherwise you would not have asked :P

Doing the following as root will make elasticsearch a one-node cluster : leafpad /etc/elasticsearch/elasticsearch.yml Uncomment : index.number_of_shards: 1 index.number_of_replicas: 0 I reboot. Wisdom from being old and stupid (as opposed to: young and stupid) : restarting the services may do it for you, but you can't be sure until you actually reboot. You just loaded that host, just f... reboot it.

However, the unassigned shards are already there. I regret to report that I have not found yet a solution better than the idiotic workaround I came with : nuke all the elasticsearch data, with the annoying byproduct of killing the dashboards. Reboot. Re-install the dashboards.

I.D.1.0.T solution. India Delta One Zero Tango.

There is a more elegant way, I just not have figured it out.

[pevma]

This is true. I am thinking if we should make the necessary adjustments to make it a one node cluster (which will fix the "yellow" warning problem) or just explain that in a some sort of a FAQ?

[michelpy]

I think if you could have a curl command or script that deletes all unassigned shards that would be great. Alternatively, a curl command that deletes all elasticsearch data except the dashboards would be great too. I have been meaning to research these but I've had other priorities.

[pevma]

ES data reset (just updated that) - https://github.com/StamusNetworks/SELKS/wiki/Reset-stats-and-logs I dont think deleting unassigned shards is so "clean" type of approach may be......

[b-u-g-s]

Thanks much Michelpy, it worked well and now all my status are green :) I was a bit nervous about the nuke command, but reinstalling the dashboard is easy enough with Pevma's link.

Pevma, would be great indeed to get the right number of node detected from the start and the elasticsearch.yml config file configured accordingly. I guess that's for folks like us who have to get all their dashboards "green" :)

[pevma]

That also is a possibility indeed - any other suggestions?

[b-u-g-s]

yup, you need to find a background picture a bit more "sexy" than that "let's talk about SELKS" ;o)p More seriously, as I am using SELKS more (and really liking it so far!) I will let you know if I think about anything else. As you can tell I come from a user point of view who knows nothing about Suricata, ELK or Scirius so my suggestions/questions are probably not really advanced yet!

[michelpy]

@b-u-g-s : I can deal with the background if Peter fixes the fonts :P

I am a SELKS idiot myself. I like it, because it saves me time building a distro of my own, a skill I do not possess. I wished it was built on CentOs instead, because I almost never install any physical hardware without a LSI hardware RAID card, and the LSI utilities works only on CentOS/RH; but again given the time at hand it has worked so far.

I have a bigger goal : connect SELKS and EXAPGP in order to have the results of SELKS feeding /32 blackholes to my network. Ideas, anyone ?

[pevma]

@michelpy: EXAPGP?

[michelpy]

@pevma: fatal error. Insufficient caffeine blood level. Power cycle the brain.

EXABGP: https://github.com/Exa-Networks/exabgp The idea is to combine EXABGP in the SELKS host to feed /32 blackholes routes to the network. Not only the BGP feed from SELKS would blackhole the contaminated addresses at the edge router, but also possibly at trusted remote sites and at the ISP, with proper communities.

[b-u-g-s]

Michelpy: I very much like your diagram. Did you convert a diagram in ASCII Art using a tool?! or you wrote it by hand over the last 2 months? ;) On top of that... I never got to hack a toilet, would love to try ;o)

[michelpy]

@b-u-g-s : it is the original ASCII art made entirely by hand. I have never seen a tool that could do as good as me with ASCII diagrams.

As of the toilet, there was some space left in the diagram. I have a Raspberry Pi project for it. You must be fluent in French to read that :P

I am opening a new thread on the SELKS google mailing list (not finished yet). https://groups.google.com/forum/#!forum/selks

[b-u-g-s]

Michelpy: Your Ascii art skills are impressive :) I am still trying to find something to do with my Raspberry Pi, controlling a toilet sounds like a fun project!

So what is that Google Group? is that a preferred location to request features about SELKS compare to post something here on github?

[michelpy]

@b-u-g-s: here on github is for issues, and the BGP integration is not; so I moved it there. As soon as I'm done with that ExaBGP thing, I'll work on the raspi ;-)