Closed south-devel closed 5 years ago
Which Scirius version are you using ? (latest from git?)
Latest releases. (Silky Squirrel:https://github.com/StamusNetworks/scirius/archive/scirius-2.0.1.tar.gz) Also I'm using Elasticsearch:6.3.2.
Does the graph become better once you have more rules or you adjust the time span from 24 to 6hrs etc .. ?
I have collected logs more than 5days and it have 3858 alerts from 21289 Rules.
Also I did change time span to other times but it still didn't work....
ok - i had a similar issue , I think this is fixed in the next version. Would try to release on next week maybe ? CC @regit
Sure, Thanks for making scirius! :)
What is your “host” variable in /etc/scirius/local_settings ?
-- Regards, Peter Manev
On 1 Sep 2018, at 14:49, nkh8212 notifications@github.com wrote:
Sure, Thanks for making scirius! :)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
I set that in settings.py. Because I accidentally remove local_settings.py.... lol And my system's network hostname is "Home". My settings.py shown below.
USE_ELASTICSEARCH = True
#ELASTICSEARCH_ADDRESS = "127.0.0.1:9200"
ELASTICSEARCH_ADDRESS = "192.168.2.200:9200"
# You can use a star to avoid timestamping expansion for example 'logstash-*'
ELASTICSEARCH_LOGSTASH_INDEX = "logstash-*"
# You can change following value if you have different indexes for stats and alerts
ELASTICSEARCH_LOGSTASH_ALERT_INDEX = ELASTICSEARCH_LOGSTASH_INDEX
# use hourly, daily to indicate the logstash index building recurrence
ELASTICSEARCH_LOGSTASH_TIMESTAMPING = "daily"
# version of Elasticsearch
ELASTICSEARCH_VERSION = 6
# Extension used for complete field (usually "raw" or "keyword")
ELASTICSEARCH_KEYWORD = "keyword"
# Hostname field (usually "hostname" or "host")
ELASTICSEARCH_HOSTNAME = "host"
But http://192.168.2.200:8000/rules/es?query=rules_per_category&from_date=1534989859137&hosts=*
also not worked.
So I think host variable is not problem.
This is what a default one in SELKS looks like. Can you try adding it in and see if any difference after restart:
root@SELKS:~# cat /etc/scirius/local_settings.py
"""
Django settings for scirius project.
For more information on this file, see
https://docs.djangoproject.com/en/1.6/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.6/ref/settings/
"""
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = "/var/lib/scirius/"
GIT_SOURCES_BASE_DIRECTORY = os.path.join(BASE_DIR, 'git-sources/')
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
# FIXME: generate this
SECRET_KEY = 'p7o6%vq))7h3li08c%k3id(wwo*u(^dbdmx2tv#t(tb2pr9@n-'
USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "localhost:9200"
ELASTICSEARCH_VERSION = 5
KIBANA_VERSION=4
KIBANA_INDEX = ".kibana"
KIBANA_URL = "http://localhost:5601"
KIBANA_DASHBOARDS_PATH = "/opt/selks/kibana5-dashboards/"
USE_KIBANA = True
KIBANA_PROXY = True
#SURICATA_UNIX_SOCKET = "/var/run/suricata/suricata-command.socket"
USE_EVEBOX = True
EVEBOX_ADDRESS = "localhost:5636"
USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True
STATIC_ROOT="/var/lib/scirius/static/"
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db', 'db.sqlite3'),
}
}
DBBACKUP_STORAGE_OPTIONS = {'location': '/var/backups/'}
ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-*"
SURICATA_NAME_IS_HOSTNAME = True
ALLOWED_HOSTS=["*"]
root@SELKS:~#
I added local_settings.py that you commented but still didn't work...T.T
You restarted Scirius - correct ? (just double checking)
Of course I started it again every time when changing local_settings.py.
I too have this issue:
I am using the latest ELK stack: 6.4.1, I am also using the KTS6 templates from @pevma .
This is my logstash config:
input {
file {
path => ["/mnt/suricata/eve.json"]
#sincedb_path => ["/var/lib/logstash/"]
sincedb_path => ["/etc/logstash/.sincedb_suricata"]
#start_position => beginning
codec => json
type => "Suricata"
}
}
filter {
if [type] == "Suricata" {
date {
match => [ "timestamp", "ISO8601" ]
}
ruby {
code => "
if event.get('[event_type]') == 'fileinfo'
event.set('[fileinfo][type]', event.get('[fileinfo][magic]').to_s.split(',')[0])
end
"
}
ruby {
code => "
if event.get('[event_type]') == 'alert'
sp = event.get('[alert][signature]').to_s.split(' group ')
if (sp.length == 2) and /\A\d+\z/.match(sp[1])
event.set('[alert][signature]', sp[0])
end
end
"
}
metrics {
meter => [ "eve_insert" ]
add_tag => "metric"
flush_interval => 30
}
}
if [http] {
useragent {
source => "[http][http_user_agent]"
target => "[http][user_agent]"
}
}
if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float_eu" ]
}
if ![geoip.ip] {
if [dest_ip] {
geoip {
source => "dest_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float_eu" ]
}
}
}
}
}
output {
if [event_type] and [event_type] != 'stats' {
elasticsearch {
hosts => "127.0.0.1"
index => "logstash-%{event_type}-%{+YYYY.MM.dd}"
template => "/etc/logstash/elasticsearch6-template.json"
}
} else {
elasticsearch {
hosts => "127.0.0.1"
index => "logstash-%{+YYYY.MM.dd}"
template => "/etc/logstash/elasticsearch6-template.json"
}
}
}
The time on all 3 machines (ELK is running in docker) is the same.
Also checked the host:
Can you please use Firebug or Ctrl+Shit+J
(Chrome) and refresh that page. Try looking into the network
tab of the debug mode page and send us any info/screenshots you find please.
Hi @pevma
I switched to the latest release of scirius (scirius-2.0.1)
On the rule page I also dont get any data:
These are my local_settings.py:
import os
DBBACKUP_STORAGE = 'dbbackup.storage.filesystem_storage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/opt/scirius/backups/'}
USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "192.168.1.45:9200"
ELASTICSEARCH_VERSION = 6
USE_KIBANA = True
KIBANA_URL = "http://monitoring.corp.pootenterprise.com:5601/"
KIBANA_VERSION=6
KIBANA_DASHBOARDS_PATH = "/opt/KTS6/"
USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True
ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"
ALLOWED_HOSTS = ["ips.corp.pootenterprise.com"]
can you try
SURICATA_NAME_IS_HOSTNAME = True
ALLOWED_HOSTS=["*"]
ELASTICSEARCH_KEYWORD = "keyword"
Instead of
ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"
ALLOWED_HOSTS = ["ips.corp.pootenterprise.com"]
Hi @pevma ,
With the following config it also shows the IP adresses related to a rule, which works because of the ELASTICSEARCH_KEYWORD variable. Without ELASTICSEARCH_KEYWORD it wont show anything on the rule page.
However the time related data is still empty, when I remove ELASTICSEARCH_LOGSTASH_ALERT_INDEX I dont have any data at all.
import os
DBBACKUP_STORAGE = 'dbbackup.storage.filesystem_storage'
DBBACKUP_STORAGE_OPTIONS = {'location': '/opt/scirius/backups/'}
USE_ELASTICSEARCH = True
ELASTICSEARCH_ADDRESS = "192.168.1.45:9200"
ELASTICSEARCH_VERSION = 6
USE_KIBANA = True
KIBANA_URL = "http://monitoring.corp.pootenterprise.com:5601/"
KIBANA_VERSION=6
KIBANA_DASHBOARDS_PATH = "/opt/KTS6/"
USE_SURICATA_STATS = True
USE_LOGSTASH_STATS = True
SURICATA_NAME_IS_HOSTNAME = True
ALLOWED_HOSTS=["*"]
ELASTICSEARCH_KEYWORD = "keyword"
ELASTICSEARCH_LOGSTASH_ALERT_INDEX="logstash-alert-"
Hello @Eagleman7 ,
Can you paste a screenshot of the "Response" tab of the developer view for the requests that fails ? (same screenshot that you posted before, but on the Response tab https://github.com/StamusNetworks/scirius/issues/151#issuecomment-427058071)
Just to be sure, are all the machines clock synced (mainly browser and scirius) ?
Here you go @biolds
{"from_date": "1539103765486", "42a3fb19ccaf": {"entries": [{"count": 51, "time": 1539103392000}, {"count": 73, "time": 1539105120000}, {"count": 65, "time": 1539106848000}, {"count": 71, "time": 1539108576000}, {"count": 65, "time": 1539110304000}, {"count": 76, "time": 1539112032000}, {"count": 59, "time": 1539113760000}, {"count": 78, "time": 1539115488000}, {"count": 60, "time": 1539117216000}, {"count": 78, "time": 1539118944000}, {"count": 58, "time": 1539120672000}, {"count": 67, "time": 1539122400000}, {"count": 75, "time": 1539124128000}, {"count": 67, "time": 1539125856000}, {"count": 68, "time": 1539127584000}, {"count": 67, "time": 1539129312000}, {"count": 69, "time": 1539131040000}, {"count": 67, "time": 1539132768000}, {"count": 70, "time": 1539134496000}, {"count": 67, "time": 1539136224000}, {"count": 70, "time": 1539137952000}, {"count": 68, "time": 1539139680000}, {"count": 72, "time": 1539141408000}, {"count": 63, "time": 1539143136000}, {"count": 75, "time": 1539144864000}, {"count": 67, "time": 1539146592000}, {"count": 74, "time": 1539148320000}, {"count": 63, "time": 1539150048000}, {"count": 81, "time": 1539151776000}, {"count": 61, "time": 1539153504000}, {"count": 78, "time": 1539155232000}, {"count": 60, "time": 1539156960000}, {"count": 78, "time": 1539158688000}, {"count": 63, "time": 1539160416000}, {"count": 76, "time": 1539162144000}, {"count": 59, "time": 1539163872000}, {"count": 67, "time": 1539165600000}, {"count": 69, "time": 1539167328000}, {"count": 68, "time": 1539169056000}, {"count": 74, "time": 1539170784000}, {"count": 66, "time": 1539172512000}, {"count": 73, "time": 1539174240000}, {"count": 65, "time": 1539175968000}, {"count": 72, "time": 1539177696000}, {"count": 66, "time": 1539179424000}, {"count": 71, "time": 1539181152000}, {"count": 69, "time": 1539182880000}, {"count": 77, "time": 1539184608000}, {"count": 65, "time": 1539186336000}, {"count": 71, "time": 1539188064000}, {"count": 62, "time": 1539189792000}, {"count": 74, "time": 1539191520000}, {"count": 64, "time": 1539193248000}, {"count": 74, "time": 1539194976000}, {"count": 60, "time": 1539196704000}, {"count": 79, "time": 1539198432000}, {"count": 57, "time": 1539200160000}, {"count": 78, "time": 1539201888000}, {"count": 58, "time": 1539203616000}, {"count": 78, "time": 1539205344000}, {"count": 58, "time": 1539207072000}, {"count": 66, "time": 1539208800000}, {"count": 71, "time": 1539210528000}, {"count": 65, "time": 1539212256000}, {"count": 69, "time": 1539213984000}, {"count": 66, "time": 1539215712000}, {"count": 72, "time": 1539217440000}, {"count": 65, "time": 1539219168000}, {"count": 71, "time": 1539220896000}, {"count": 66, "time": 1539222624000}, {"count": 73, "time": 1539224352000}, {"count": 66, "time": 1539226080000}, {"count": 75, "time": 1539227808000}, {"count": 65, "time": 1539229536000}, {"count": 77, "time": 1539231264000}, {"count": 68, "time": 1539232992000}, {"count": 70, "time": 1539234720000}, {"count": 63, "time": 1539236448000}, {"count": 78, "time": 1539238176000}, {"count": 61, "time": 1539239904000}, {"count": 78, "time": 1539241632000}, {"count": 61, "time": 1539243360000}, {"count": 76, "time": 1539245088000}, {"count": 60, "time": 1539246816000}, {"count": 77, "time": 1539248544000}, {"count": 59, "time": 1539250272000}, {"count": 65, "time": 1539252000000}, {"count": 71, "time": 1539253728000}, {"count": 66, "time": 1539255456000}, {"count": 71, "time": 1539257184000}, {"count": 66, "time": 1539258912000}, {"count": 72, "time": 1539260640000}, {"count": 67, "time": 1539262368000}, {"count": 69, "time": 1539264096000}, {"count": 67, "time": 1539265824000}, {"count": 70, "time": 1539267552000}, {"count": 66, "time": 1539269280000}, {"count": 70, "time": 1539271008000}, {"count": 65, "time": 1539272736000}, {"count": 73, "time": 1539274464000}, {"count": 11, "time": 1539276192000}]}, "interval": 1728000}
So it looks like it is getting numbers from the time range and the rule IDs. But its not showing them.
Solved on IRC, the issue was the SURICATA_NAME_IS_HOSTNAME had to be set to False, because the suricata hostname set in scirius is different than the hostname of the host.
Can you confirm this also works for you @nkh8212
hmm... It not work for me. T.T.... But @Eagleman7 confirmed It work. I'm going to try fix it. Thanks for help :)
And It query is -> http://192.168.2.200:8000/rules/es?query=rules_per_category&from_date=1534989859137&hosts=Home
And It return
What am i wrong?