Alerts activity not working.

[south-devel]

And It query is -> http://192.168.2.200:8000/rules/es?query=rules_per_category&from_date=1534989859137&hosts=Home

And It return

• {"children": [], "key": "categories"}

What am i wrong?

[pevma]

Which Scirius version are you using ? (latest from git?)

[south-devel]

Latest releases. (Silky Squirrel:https://github.com/StamusNetworks/scirius/archive/scirius-2.0.1.tar.gz) Also I'm using Elasticsearch:6.3.2.

[pevma]

Does the graph become better once you have more rules or you adjust the time span from 24 to 6hrs etc .. ?

[south-devel]

I have collected logs more than 5days and it have 3858 alerts from 21289 Rules.

Also I did change time span to other times but it still didn't work....

[pevma]

ok - i had a similar issue , I think this is fixed in the next version. Would try to release on next week maybe ? CC @regit

[south-devel]

Sure, Thanks for making scirius! :)

[pevma]

What is your “host” variable in /etc/scirius/local_settings ?

-- Regards, Peter Manev

On 1 Sep 2018, at 14:49, nkh8212 notifications@github.com wrote:

Sure, Thanks for making scirius! :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[south-devel]

I set that in settings.py. Because I accidentally remove local_settings.py.... lol And my system's network hostname is "Home". My settings.py shown below.

USE_ELASTICSEARCH = True #ELASTICSEARCH_ADDRESS = "127.0.0.1:9200" ELASTICSEARCH_ADDRESS = "192.168.2.200:9200" # You can use a star to avoid timestamping expansion for example 'logstash-*' ELASTICSEARCH_LOGSTASH_INDEX = "logstash-*" # You can change following value if you have different indexes for stats and alerts ELASTICSEARCH_LOGSTASH_ALERT_INDEX = ELASTICSEARCH_LOGSTASH_INDEX # use hourly, daily to indicate the logstash index building recurrence ELASTICSEARCH_LOGSTASH_TIMESTAMPING = "daily" # version of Elasticsearch ELASTICSEARCH_VERSION = 6 # Extension used for complete field (usually "raw" or "keyword") ELASTICSEARCH_KEYWORD = "keyword" # Hostname field (usually "hostname" or "host") ELASTICSEARCH_HOSTNAME = "host"

But http://192.168.2.200:8000/rules/es?query=rules_per_category&from_date=1534989859137&hosts=* also not worked. So I think host variable is not problem.

[pevma]

This is what a default one in SELKS looks like. Can you try adding it in and see if any difference after restart:

root@SELKS:~# cat /etc/scirius/local_settings.py 
"""
Django settings for scirius project.

For more information on this file, see
https://docs.djangoproject.com/en/1.6/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.6/ref/settings/
"""

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = "/var/lib/scirius/"
GIT_SOURCES_BASE_DIRECTORY = os.path.join(BASE_DIR, 'git-sources/')

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
# FIXME: generate this
SECRET_KEY = 'p7o6%vq))7h3li08c%k3id(wwo*u(^dbdmx2tv#t(tb2pr9@n-'
USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "localhost:9200"
ELASTICSEARCH_VERSION = 5
KIBANA_VERSION=4
KIBANA_INDEX = ".kibana"
KIBANA_URL = "http://localhost:5601"
KIBANA_DASHBOARDS_PATH = "/opt/selks/kibana5-dashboards/"
USE_KIBANA = True
KIBANA_PROXY = True

#SURICATA_UNIX_SOCKET = "/var/run/suricata/suricata-command.socket"

USE_EVEBOX = True
EVEBOX_ADDRESS = "localhost:5636"

USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True
STATIC_ROOT="/var/lib/scirius/static/"

DATABASES = {
  'default': {
     'ENGINE': 'django.db.backends.sqlite3',
     'NAME': os.path.join(BASE_DIR, 'db', 'db.sqlite3'),
  }
}
DBBACKUP_STORAGE_OPTIONS = {'location': '/var/backups/'}

ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-*"

SURICATA_NAME_IS_HOSTNAME = True

ALLOWED_HOSTS=["*"]
root@SELKS:~# 

[south-devel]

I added local_settings.py that you commented but still didn't work...T.T

[pevma]

You restarted Scirius - correct ? (just double checking)

[south-devel]

Of course I started it again every time when changing local_settings.py.

[Eagleman7]

I too have this issue:

I am using the latest ELK stack: 6.4.1, I am also using the KTS6 templates from @pevma .

This is my logstash config:

input {
  file {
    path => ["/mnt/suricata/eve.json"]
    #sincedb_path => ["/var/lib/logstash/"]
    sincedb_path => ["/etc/logstash/.sincedb_suricata"]
        #start_position => beginning
    codec =>   json
    type => "Suricata"
  }

}

filter {
  if [type] == "Suricata" {

    date {
      match => [ "timestamp", "ISO8601" ]
    }

    ruby {
      code => "
        if event.get('[event_type]') == 'fileinfo'
          event.set('[fileinfo][type]', event.get('[fileinfo][magic]').to_s.split(',')[0])
        end
      "
    }
    ruby {
      code => "
        if event.get('[event_type]') == 'alert'
          sp = event.get('[alert][signature]').to_s.split(' group ')
          if (sp.length == 2) and /\A\d+\z/.match(sp[1])
            event.set('[alert][signature]', sp[0])
          end
        end
      "
     }

    metrics {
      meter => [ "eve_insert" ]
      add_tag => "metric"
      flush_interval => 30
    }
  }

  if [http] {
    useragent {
       source => "[http][http_user_agent]"
       target => "[http][user_agent]"
    }
  }
  if [src_ip]  {
    geoip {
      source => "src_ip"
      target => "geoip"
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float_eu" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip"
          target => "geoip"
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float_eu" ]
        }
      }
    }
  }
}

output {
  if [event_type] and [event_type] != 'stats' {
    elasticsearch {
      hosts => "127.0.0.1"
      index => "logstash-%{event_type}-%{+YYYY.MM.dd}"
      template => "/etc/logstash/elasticsearch6-template.json"
    }
  } else {
    elasticsearch {
      hosts => "127.0.0.1"
      index => "logstash-%{+YYYY.MM.dd}"
      template => "/etc/logstash/elasticsearch6-template.json"
    }
  }
}

The time on all 3 machines (ELK is running in docker) is the same.

Also checked the host:

[pevma]

Can you please use Firebug or Ctrl+Shit+J (Chrome) and refresh that page. Try looking into the network tab of the debug mode page and send us any info/screenshots you find please.

[Eagleman7]

Hi @pevma

I switched to the latest release of scirius (scirius-2.0.1)

[Eagleman7]

On the rule page I also dont get any data:

[Eagleman7]

These are my local_settings.py:

import os

DBBACKUP_STORAGE = 'dbbackup.storage.filesystem_storage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/opt/scirius/backups/'}

USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "192.168.1.45:9200"
ELASTICSEARCH_VERSION = 6

USE_KIBANA = True
KIBANA_URL = "http://monitoring.corp.pootenterprise.com:5601/"
KIBANA_VERSION=6

KIBANA_DASHBOARDS_PATH = "/opt/KTS6/"

USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True

ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"

ALLOWED_HOSTS = ["ips.corp.pootenterprise.com"]

[pevma]

can you try

SURICATA_NAME_IS_HOSTNAME = True
ALLOWED_HOSTS=["*"]
ELASTICSEARCH_KEYWORD = "keyword"

Instead of

ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"

ALLOWED_HOSTS = ["ips.corp.pootenterprise.com"]

[Eagleman7]

Hi @pevma ,

With the following config it also shows the IP adresses related to a rule, which works because of the ELASTICSEARCH_KEYWORD variable. Without ELASTICSEARCH_KEYWORD it wont show anything on the rule page.

However the time related data is still empty, when I remove ELASTICSEARCH_LOGSTASH_ALERT_INDEX I dont have any data at all.

import os

DBBACKUP_STORAGE = 'dbbackup.storage.filesystem_storage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/opt/scirius/backups/'}

USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "192.168.1.45:9200"
ELASTICSEARCH_VERSION = 6

USE_KIBANA = True
KIBANA_URL = "http://monitoring.corp.pootenterprise.com:5601/"
KIBANA_VERSION=6

KIBANA_DASHBOARDS_PATH = "/opt/KTS6/"

USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True

SURICATA_NAME_IS_HOSTNAME = True
ALLOWED_HOSTS=["*"]
ELASTICSEARCH_KEYWORD = "keyword"

ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"

[biolds]

Hello @Eagleman7 ,

Can you paste a screenshot of the "Response" tab of the developer view for the requests that fails ? (same screenshot that you posted before, but on the Response tab https://github.com/StamusNetworks/scirius/issues/151#issuecomment-427058071)

Just to be sure, are all the machines clock synced (mainly browser and scirius) ?

[Eagleman7]

Here you go @biolds

http://ips.corp.pootenterprise.com:8000/rules/es?query=timeline&from_date=1539103765486&hosts=ips.corp.pootenterprise.com

{"from_date": "1539103765486", "42a3fb19ccaf": {"entries": [{"count": 51, "time": 1539103392000}, {"count": 73, "time": 1539105120000}, {"count": 65, "time": 1539106848000}, {"count": 71, "time": 1539108576000}, {"count": 65, "time": 1539110304000}, {"count": 76, "time": 1539112032000}, {"count": 59, "time": 1539113760000}, {"count": 78, "time": 1539115488000}, {"count": 60, "time": 1539117216000}, {"count": 78, "time": 1539118944000}, {"count": 58, "time": 1539120672000}, {"count": 67, "time": 1539122400000}, {"count": 75, "time": 1539124128000}, {"count": 67, "time": 1539125856000}, {"count": 68, "time": 1539127584000}, {"count": 67, "time": 1539129312000}, {"count": 69, "time": 1539131040000}, {"count": 67, "time": 1539132768000}, {"count": 70, "time": 1539134496000}, {"count": 67, "time": 1539136224000}, {"count": 70, "time": 1539137952000}, {"count": 68, "time": 1539139680000}, {"count": 72, "time": 1539141408000}, {"count": 63, "time": 1539143136000}, {"count": 75, "time": 1539144864000}, {"count": 67, "time": 1539146592000}, {"count": 74, "time": 1539148320000}, {"count": 63, "time": 1539150048000}, {"count": 81, "time": 1539151776000}, {"count": 61, "time": 1539153504000}, {"count": 78, "time": 1539155232000}, {"count": 60, "time": 1539156960000}, {"count": 78, "time": 1539158688000}, {"count": 63, "time": 1539160416000}, {"count": 76, "time": 1539162144000}, {"count": 59, "time": 1539163872000}, {"count": 67, "time": 1539165600000}, {"count": 69, "time": 1539167328000}, {"count": 68, "time": 1539169056000}, {"count": 74, "time": 1539170784000}, {"count": 66, "time": 1539172512000}, {"count": 73, "time": 1539174240000}, {"count": 65, "time": 1539175968000}, {"count": 72, "time": 1539177696000}, {"count": 66, "time": 1539179424000}, {"count": 71, "time": 1539181152000}, {"count": 69, "time": 1539182880000}, {"count": 77, "time": 1539184608000}, {"count": 65, "time": 1539186336000}, {"count": 71, "time": 1539188064000}, {"count": 62, "time": 1539189792000}, {"count": 74, "time": 1539191520000}, {"count": 64, "time": 1539193248000}, {"count": 74, "time": 1539194976000}, {"count": 60, "time": 1539196704000}, {"count": 79, "time": 1539198432000}, {"count": 57, "time": 1539200160000}, {"count": 78, "time": 1539201888000}, {"count": 58, "time": 1539203616000}, {"count": 78, "time": 1539205344000}, {"count": 58, "time": 1539207072000}, {"count": 66, "time": 1539208800000}, {"count": 71, "time": 1539210528000}, {"count": 65, "time": 1539212256000}, {"count": 69, "time": 1539213984000}, {"count": 66, "time": 1539215712000}, {"count": 72, "time": 1539217440000}, {"count": 65, "time": 1539219168000}, {"count": 71, "time": 1539220896000}, {"count": 66, "time": 1539222624000}, {"count": 73, "time": 1539224352000}, {"count": 66, "time": 1539226080000}, {"count": 75, "time": 1539227808000}, {"count": 65, "time": 1539229536000}, {"count": 77, "time": 1539231264000}, {"count": 68, "time": 1539232992000}, {"count": 70, "time": 1539234720000}, {"count": 63, "time": 1539236448000}, {"count": 78, "time": 1539238176000}, {"count": 61, "time": 1539239904000}, {"count": 78, "time": 1539241632000}, {"count": 61, "time": 1539243360000}, {"count": 76, "time": 1539245088000}, {"count": 60, "time": 1539246816000}, {"count": 77, "time": 1539248544000}, {"count": 59, "time": 1539250272000}, {"count": 65, "time": 1539252000000}, {"count": 71, "time": 1539253728000}, {"count": 66, "time": 1539255456000}, {"count": 71, "time": 1539257184000}, {"count": 66, "time": 1539258912000}, {"count": 72, "time": 1539260640000}, {"count": 67, "time": 1539262368000}, {"count": 69, "time": 1539264096000}, {"count": 67, "time": 1539265824000}, {"count": 70, "time": 1539267552000}, {"count": 66, "time": 1539269280000}, {"count": 70, "time": 1539271008000}, {"count": 65, "time": 1539272736000}, {"count": 73, "time": 1539274464000}, {"count": 11, "time": 1539276192000}]}, "interval": 1728000}

So it looks like it is getting numbers from the time range and the rule IDs. But its not showing them.

[biolds]

Solved on IRC, the issue was the SURICATA_NAME_IS_HOSTNAME had to be set to False, because the suricata hostname set in scirius is different than the hostname of the host.

[Eagleman7]

Can you confirm this also works for you @nkh8212

[south-devel]

hmm... It not work for me. T.T.... But @Eagleman7 confirmed It work. I'm going to try fix it. Thanks for help :)