search

desaster / kippo

Kippo - SSH Honeypot
1.62k stars 277 forks source link

Add ElasticSearch support to Kippo #135

Open ikoniaris opened 10 years ago

ikoniaris commented 10 years ago

Hi desaster.

This pull request adds ElasticSearch support to Kippo.

Configuration: I have added a new section to the config file where the ES instance/cluster's values should be set.

Logging module: I have added a new file named "elasticsearch.py" into "kippo/dblog/" in the same fashion as the other logging modules and followed the same function overriding style.

Testing: I have tested it myself and it works correctly. The login attempts can be visualized with Kibana, like in Kippo2ElasticSearch's case: http://bruteforce.gr/kippo2elasticsearch (see the screenshots). If you don't want to mess with installing ELK, you can use the latest version of HoneyDrive (released today!) to test it yourself: http://bruteforce.gr/honeydrive

Additional requirements:

  1. pyes: https://pypi.python.org/pypi/pyes
  2. GeoIP: https://pypi.python.org/pypi/GeoIP
  3. A GeoIP.dat database from MaxMind: http://dev.maxmind.com/geoip/legacy/geolite/

1 and 2 are easily installable via pip. Regarding number 3 above, I have included a database in the pull request that I use to initialize the geoip object I use for geolocating the remote IP. You can skip it if you like though and add an extra configuration option to kippo.cfg for the user to select whether to enable geolocation or not. If yes, he will have to manually download the GeoIP.dat file and unzip it in the root folder for example. By the way, a requirements.txt file should also be added at some point to track those packages.

Extras: I've also included a more proper .gitignore file. I should have opened another pull request for this perhaps but I was coding the ElasticSearch part at the same time.

Let me know what you think.

ikoniaris commented 10 years ago

This also solves: https://github.com/desaster/kippo/issues/138

desaster commented 10 years ago

Thanks for your pull request!

I've done some quick tests with the help of honeydrive;

My concern here is the same as with the hpfeeds logger, the lack of asynchronous operation. In case of connection problems, the entire operation of the application is blocked.

Simulated with:

sudo iptables -A OUTPUT -p tcp --dport 9200 -j DROP

If the underlying library doesn't allow for asynchronous operation, perhaps we can use twisted.internet.threads.deferToThread to work around it.

I also roughly tested how it manages a disappearing elasticsearch, and while there are nasty errors in the console, the honeypot seems to continue operation fine.

ikoniaris commented 10 years ago

Hi @desaster, thanks a lot for testing this!

I understand your concern, blocking operations can be problematic. Your suggestion seems fine to me. Although I don't have any experience with the Twisted library I can take a look and see what can be done. Unless you're interested in trying yourself.

I've also tested what happens if the ES service goes down, and had the same experience. The fact that the honeypot continues working is "good", although masking of the errors is probably needed.

desaster commented 10 years ago

there seems to be some existing work on making elasticsearch work with twisted, such as this https://github.com/jkoelker/txes

however this is just after a quick google search, i haven't looked at how they work yet.

ikoniaris commented 10 years ago

There is also this Tornado based library for async communication with ES: https://github.com/globocom/tornado-es