Updates ELK Install to use 7.x versions - Ref #740

[erwanlr]

Script updated and working.

A few things:

• Make sure the system MHN is installed on has enough RAM and disk space. 3GB of RAM and 10GB of disk seem to be the minimum (tested with only one Sensor though). For instance, MHN installed on an Ubuntu server with only the basic (mail and SSH) along with ELK takes around 8.8GB of disk.
• I was unable to fix the following warnings in Logstash:
  • Detected a 6.x and above cluster: thetypeevent field won't be used to determine the document _type {:es_version=>7}. Not sure where this type is taken from, maybe from the log.
  • [org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization.
• It seems that HPFeeds (or something in between) restricts the data written in the mhn-json.log log which is then parsed by Logstash and fed to ES. For instance, even though the below was sent via hp_feeds

  // I, [2020-01-05T14:39:45.802598 #81583]  INFO -- : publish to wordpot.events: 
  {  
  "source_ip":"192.168.1.42",
  "source_port":9292,
  "dest_ip":"192.168.1.42",
  "dest_port":"9292",
  "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0",
  "url":"/wp-login.php",
  "request":{  
      "method":"POST",
      "path":"/wp-login.php",
      "query_string":{},
      "body":{  
          "log":"gwefery",
          "pwd":"gfurg",
          "wp-submit":"Log In",
          "redirect_to":"/wp-admin/",
          "testcookie":"1"
      },
      "referer":"http://192.168.1.42:9292/wp-login.php"
    }
  }

  the data in mhn-json.log was missing the request and user_agent data:

  {  
  "direction":"inbound",
  "protocol":"ip",
  "ids_type":"network",
  "timestamp":"2020-01-05T13:39:45.943063",
  "vendor_product":"Wordpot",
  "type":"wordpot.alerts",
  "app":"wordpot",
  "src_ip":"192.168.1.42",
  "dest_port":"9292",
  "signature":"Wordpress Exploit, Scan, or Enumeration Attempted",
  "src_port":9292,
  "request_url":"/wp-login.php",
  "dest_ip":"192.168.1.42",
  "sensor":"0cad2d40-2fb8-11ea-8b4f-0800271514ba",
  "transport":"tcp",
  "severity":"high"
  }

  So if you have any idea how to be able to get the 'custom' data received by HPFeeds written in the log, please share

[d1str0]

hpfeeds-logger is the package that normalizes the honeypot logs. That package will need to be updated if you want extra data.

[d1str0]

@erwanlr so by default, remote machines won't be able to connect to kibana because it binds to localhost, not it's IP or 0.0.0.0

[d1str0]

@erwanlr if you can change the kibana config so that it binds to 0.0.0.0, that'd be great. Everything else looks good and I'll merge once this change is made.

[erwanlr]

Yes, by default all ELK runs on localhost, and services have to be exposed if needed.

I would not bind Kibana to 0.0.0.0, as there is no login form. It can be configured though it seems (https://www.elastic.co/guide/en/kibana/current/kibana-authentication.html) but I haven't tried that.

[d1str0]

Good point. Hmmm. We need to either add an option to bind to 0.0.0.0, or a descriptive notice for users.

[erwanlr]

Notice added

[d1str0]

Beautiful.