StamusNetworks / SELKS

A Suricata based IDS/IPS/NSM distro
https://www.stamus-networks.com/open-source/#selks
GNU General Public License v3.0
1.38k stars 278 forks source link

No data for period - Problem! #186

Open Mauro2k opened 5 years ago

Mauro2k commented 5 years ago

I have made a clean installation in an ESXi, I have configured everything according to instructions. I see that there are packages received by the EVENTBOX, but I do not have data in the main Dashboard.

I have already configured my networks and interfaces but I have not been successful.

Can someone guide me with this topic?

attach a series of captures

ids5 4ids 3ids ids2 ids1

pevma commented 5 years ago

What is the output of selks-healthcheck_stamus?

Mauro2k commented 5 years ago

Output selks_healthcheck_stamus:

selks-user@SELKS:~$ selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:59 -03; 49min ago Docs: man:systemd-sysv-generator(8) Process: 661 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 10 (limit: 4915) CGroup: /system.slice/suricata.service └─873 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:59 -03; 49min ago Docs: http://www.elastic.co Main PID: 666 (java) Tasks: 82 (limit: 4915) CGroup: /system.slice/elasticsearch.service ├─ 666 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOcc…et └─1090 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:57 -03; 49min ago Main PID: 415 (java) Tasks: 36 (limit: 4915) CGroup: /system.slice/logstash.service └─415 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccu…sh ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:58 -03; 49min ago Main PID: 487 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/kibana.service └─487 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli …ml ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:58 -03; 49min ago Main PID: 483 (evebox) Tasks: 13 (limit: 4915) CGroup: /system.slice/evebox.service └─483 /usr/bin/evebox server ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2019-05-24 15:10:50 -03; 42min ago Process: 1432 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 1432 (code=exited, status=1/FAILURE) ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:06:42 -03; 46min ago Main PID: 1296 (sh) Tasks: 5 (limit: 4915) CGroup: /system.slice/molochpcapread-selks.service ├─1296 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ >> /data/moloch/log…&1 └─1297 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ error: <class 'socket.error'>, [Errno 13] Permission denied: file: /usr/lib/python2.7/socket.py line: 228 ii elasticsearch 6.8.0 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.7.6 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.0 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.0-1 all An extensible logging pipeline ii moloch 1.8.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 2019040702-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 4.9G 0 4.9G 0% /dev tmpfs tmpfs 1001M 17M 984M 2% /run /dev/sda1 ext4 334G 6.9G 310G 3% / tmpfs tmpfs 4.9G 0 4.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup tmpfs tmpfs 1001M 4.0K 1001M 1% /run/user/112 tmpfs tmpfs 1001M 0 1001M 0% /run/user/1000

pevma commented 5 years ago

Ok thanks for the feedback. Are there events on the “SN ALL” dashboard ? And are there alerts in “SN ALERTS” dashboard ?

-- Regards, Peter Manev

On 24 May 2019, at 20:54, Mac notifications@github.com wrote:

Output selks_healthcheck_stamus:

selks-user@SELKS:~$ selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:59 -03; 49min ago Docs: man:systemd-sysv-generator(8) Process: 661 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 10 (limit: 4915) CGroup: /system.slice/suricata.service └─873 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:59 -03; 49min ago Docs: http://www.elastic.co Main PID: 666 (java) Tasks: 82 (limit: 4915) CGroup: /system.slice/elasticsearch.service ├─ 666 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOcc…et └─1090 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:57 -03; 49min ago Main PID: 415 (java) Tasks: 36 (limit: 4915) CGroup: /system.slice/logstash.service └─415 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccu…sh ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:58 -03; 49min ago Main PID: 487 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/kibana.service └─487 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli …ml ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:03:58 -03; 49min ago Main PID: 483 (evebox) Tasks: 13 (limit: 4915) CGroup: /system.slice/evebox.service └─483 /usr/bin/evebox server ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2019-05-24 15:10:50 -03; 42min ago Process: 1432 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 1432 (code=exited, status=1/FAILURE) ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-05-24 15:06:42 -03; 46min ago Main PID: 1296 (sh) Tasks: 5 (limit: 4915) CGroup: /system.slice/molochpcapread-selks.service ├─1296 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ >> /data/moloch/log…&1 └─1297 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ error: <class 'socket.error'>, [Errno 13] Permission denied: file: /usr/lib/python2.7/socket.py line: 228 ii elasticsearch 6.8.0 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.7.6 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.0 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.0-1 all An extensible logging pipeline ii moloch 1.8.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 2019040702-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 4.9G 0 4.9G 0% /dev tmpfs tmpfs 1001M 17M 984M 2% /run /dev/sda1 ext4 334G 6.9G 310G 3% / tmpfs tmpfs 4.9G 0 4.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup tmpfs tmpfs 1001M 4.0K 1001M 1% /run/user/112 tmpfs tmpfs 1001M 0 1001M 0% /run/user/1000

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Mauro2k commented 5 years ago

Only some data in SN-ALL SN-ALL SNALL

SNALERTS SNALERTS

Mauro2k commented 5 years ago

Logstash has full data for what I see in "Discover"

Discover

pevma commented 5 years ago

It seems there is traffic just no alerts. Anything that has recently changed on your network ?

Mauro2k commented 5 years ago

No, no change. What strikes me is that I have traffic, but not alerts. I have generated alerts of all kinds and they do not appear. I do not know what is happening :(

Mauro2k commented 5 years ago

In the configuration file "suricata.yaml" I have configured my vLans within HOME_NET: suricatayalm

This is fine, right?

pevma commented 5 years ago

What is the output of tail -50 /var/log/suricata/suricata.log ?

-- Regards, Peter Manev

On 24 May 2019, at 22:19, Mac notifications@github.com wrote:

No, no change. What strikes me is that I have traffic, but not alerts. I have generated alerts of all kinds and they do not appear. I do not know what is happening :(

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Mauro2k commented 5 years ago

Output:


selks-user@SELKS:/var/log/suricata$ tail -50 /var/log/suricata/suricata.log [818] 24/5/2019 -- 17:27:59 - (output-json-stats.c:468) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [818] 24/5/2019 -- 17:27:59 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- http-log output device (regular) initialized: http.log [818] 24/5/2019 -- 17:27:59 - (log-pcap.c:1312) (PcapLogInitCtx) -- Using log dir /data/nsm/ [818] 24/5/2019 -- 17:27:59 - (log-pcap.c:1423) (PcapLogInitCtx) -- Selected pcap-log compression method: (null) [818] 24/5/2019 -- 17:27:59 - (log-pcap.c:1427) (PcapLogInitCtx) -- using multi logging [818] 24/5/2019 -- 17:27:59 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- stats output device (regular) initialized: stats.log [818] 24/5/2019 -- 17:27:59 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [818] 24/5/2019 -- 17:27:59 - (reputation.c:639) (SRepInit) -- Loading reputation file: /etc/suricata/rules/scirius-iprep.list [818] 24/5/2019 -- 17:28:10 - (detect-engine-loader.c:351) (SigLoadSignatures) -- 1 rule files processed. 21778 rules successfully loaded, 0 rules failed [818] 24/5/2019 -- 17:28:10 - (util-threshold-config.c:1126) (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found [818] 24/5/2019 -- 17:28:10 - (detect-engine-build.c:1426) (SigAddressPrepareStage1) -- 21782 signatures processed. 68 are IP-only rules, 6600 are inspecting packet payload, 14996 inspect application layer, 0 are decoder event only [818] 24/5/2019 -- 17:28:30 - (util-runmodes.c:297) (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 4 thread(s) [992] 24/5/2019 -- 17:28:30 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [992] 24/5/2019 -- 17:28:30 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 15 files. [994] 24/5/2019 -- 17:28:30 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [994] 24/5/2019 -- 17:28:30 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 15 files. [995] 24/5/2019 -- 17:28:30 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [995] 24/5/2019 -- 17:28:30 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 16 files. [996] 24/5/2019 -- 17:28:30 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [996] 24/5/2019 -- 17:28:30 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 15 files. [818] 24/5/2019 -- 17:28:30 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [818] 24/5/2019 -- 17:28:30 - (unix-manager.c:131) (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket' [818] 24/5/2019 -- 17:28:30 - (tm-threads.c:2157) (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started. [996] 24/5/2019 -- 17:28:30 - (source-af-packet.c:509) (AFPPeersListReachedInc) -- All AFP capture threads are running. [818] 24/5/2019 -- 17:32:17 - (suricata.c:2827) (SuricataMainLoop) -- Signal Received. Stopping engine. [818] 24/5/2019 -- 17:32:17 - (suricata.c:1088) (SCPrintElapsedTime) -- time elapsed 226.823s [818] 24/5/2019 -- 17:32:17 - (counters.c:849) (StatsLogSummary) -- Alerts: 0 [818] 24/5/2019 -- 17:32:17 - (detect-engine-build.c:1732) (SigAddressCleanupStage1) -- cleaning up signature grouping structure... complete [818] 24/5/2019 -- 17:32:18 - (util-device.c:329) (LiveDeviceListClean) -- Stats for 'ens192': pkts: 3341, drop: 0 (0.00%), invalid chksum: 0 [728] 24/5/2019 -- 17:34:41 - (suricata.c:1064) (LogVersion) -- This is Suricata version 5.0.0-dev (rev 231496f16) [728] 24/5/2019 -- 17:34:41 - (util-cpu.c:171) (UtilCpuPrintSummary) -- CPUs/cores online: 4 [865] 24/5/2019 -- 17:34:29 - (util-privs.c:93) (SCDropMainThreadCaps) -- dropped the caps for main thread [865] 24/5/2019 -- 17:34:29 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [865] 24/5/2019 -- 17:34:29 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- fast output device (regular) initialized: fast.log [865] 24/5/2019 -- 17:34:29 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- eve-log output device (regular) initialized: eve.json [865] 24/5/2019 -- 17:34:29 - (output-json-email-common.c:455) (OutputEmailInitConf) -- Going to log the md5 sum of email body [865] 24/5/2019 -- 17:34:29 - (output-json-email-common.c:459) (OutputEmailInitConf) -- Going to log the md5 sum of email subject [865] 24/5/2019 -- 17:34:29 - (output-json-dnp3.c:392) (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. [865] 24/5/2019 -- 17:34:29 - (output-json-dnp3.c:392) (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. [865] 24/5/2019 -- 17:34:29 - (output-json-stats.c:468) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [865] 24/5/2019 -- 17:34:29 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- http-log output device (regular) initialized: http.log [865] 24/5/2019 -- 17:34:29 - (log-pcap.c:1312) (PcapLogInitCtx) -- Using log dir /data/nsm/ [865] 24/5/2019 -- 17:34:29 - (log-pcap.c:1423) (PcapLogInitCtx) -- Selected pcap-log compression method: (null) [865] 24/5/2019 -- 17:34:29 - (log-pcap.c:1427) (PcapLogInitCtx) -- using multi logging [865] 24/5/2019 -- 17:34:29 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- stats output device (regular) initialized: stats.log [865] 24/5/2019 -- 17:34:29 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [865] 24/5/2019 -- 17:34:29 - (reputation.c:639) (SRepInit) -- Loading reputation file: /etc/suricata/rules/scirius-iprep.list [865] 24/5/2019 -- 17:34:39 - (detect-engine-loader.c:351) (SigLoadSignatures) -- 1 rule files processed. 21778 rules successfully loaded, 0 rules failed [865] 24/5/2019 -- 17:34:39 - (util-threshold-config.c:1126) (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found [865] 24/5/2019 -- 17:34:40 - (detect-engine-build.c:1426) (SigAddressPrepareStage1) -- 21782 signatures processed. 68 are IP-only rules, 6600 are inspecting packet payload, 14996 inspect application layer, 0 are decoder event only


pevma commented 5 years ago

Is that the end of the file? There should be something more to the bottom i think. The network definition looks ok yes.

Mauro2k commented 5 years ago

Paste again:


selks-user@SELKS:/var/log/suricata$ tail -50 /var/log/suricata/suricata.log [1042] 26/5/2019 -- 02:01:44 - (detect-engine-loader.c:351) (SigLoadSignatures) -- 1 rule files processed. 21784 rules successfully loaded, 0 rules failed [1042] 26/5/2019 -- 02:01:44 - (util-threshold-config.c:1126) (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found [1042] 26/5/2019 -- 02:01:45 - (detect-engine-build.c:1426) (SigAddressPrepareStage1) -- 21788 signatures processed. 68 are IP-only rules, 6601 are inspecting packet payload, 15001 inspect application layer, 0 are decoder event only [1042] 26/5/2019 -- 02:02:04 - (util-runmodes.c:297) (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 4 thread(s) [1101] 26/5/2019 -- 02:02:04 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [1101] 26/5/2019 -- 02:02:04 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 18 files. [1103] 26/5/2019 -- 02:02:04 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [1103] 26/5/2019 -- 02:02:04 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 18 files. [1104] 26/5/2019 -- 02:02:04 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [1104] 26/5/2019 -- 02:02:04 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 19 files. [1105] 26/5/2019 -- 02:02:05 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [1105] 26/5/2019 -- 02:02:05 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 18 files. [1042] 26/5/2019 -- 02:02:05 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [1042] 26/5/2019 -- 02:02:05 - (unix-manager.c:131) (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket' [1042] 26/5/2019 -- 02:02:05 - (tm-threads.c:2157) (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started. [1105] 26/5/2019 -- 02:02:05 - (source-af-packet.c:509) (AFPPeersListReachedInc) -- All AFP capture threads are running. [8169] 27/5/2019 -- 02:02:04 - (suricata.c:1064) (LogVersion) -- This is Suricata version 5.0.0-dev (rev 231496f16) [8169] 27/5/2019 -- 02:02:05 - (util-cpu.c:171) (UtilCpuPrintSummary) -- CPUs/cores online: 4 [8171] 27/5/2019 -- 02:02:05 - (util-privs.c:93) (SCDropMainThreadCaps) -- dropped the caps for main thread [8171] 27/5/2019 -- 02:02:05 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [8171] 27/5/2019 -- 02:02:05 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- fast output device (regular) initialized: fast.log [8171] 27/5/2019 -- 02:02:05 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- eve-log output device (regular) initialized: eve.json [8171] 27/5/2019 -- 02:02:05 - (output-json-email-common.c:455) (OutputEmailInitConf) -- Going to log the md5 sum of email body [8171] 27/5/2019 -- 02:02:05 - (output-json-email-common.c:459) (OutputEmailInitConf) -- Going to log the md5 sum of email subject [8171] 27/5/2019 -- 02:02:05 - (output-json-dnp3.c:392) (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. [8171] 27/5/2019 -- 02:02:05 - (output-json-dnp3.c:392) (OutputDNP3LogInitSub) -- DNP3 log sub-module initialized. [8171] 27/5/2019 -- 02:02:05 - (output-json-stats.c:468) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [8171] 27/5/2019 -- 02:02:05 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- http-log output device (regular) initialized: http.log [8171] 27/5/2019 -- 02:02:05 - (log-pcap.c:1312) (PcapLogInitCtx) -- Using log dir /data/nsm/ [8171] 27/5/2019 -- 02:02:05 - (log-pcap.c:1423) (PcapLogInitCtx) -- Selected pcap-log compression method: (null) [8171] 27/5/2019 -- 02:02:05 - (log-pcap.c:1427) (PcapLogInitCtx) -- using multi logging [8171] 27/5/2019 -- 02:02:05 - (util-logopenfile.c:478) (SCConfLogOpenGeneric) -- stats output device (regular) initialized: stats.log [8171] 27/5/2019 -- 02:02:05 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [8171] 27/5/2019 -- 02:02:05 - (reputation.c:639) (SRepInit) -- Loading reputation file: /etc/suricata/rules/scirius-iprep.list [8171] 27/5/2019 -- 02:02:13 - (detect-engine-loader.c:351) (SigLoadSignatures) -- 1 rule files processed. 21782 rules successfully loaded, 0 rules failed [8171] 27/5/2019 -- 02:02:13 - (util-threshold-config.c:1126) (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found [8171] 27/5/2019 -- 02:02:14 - (detect-engine-build.c:1426) (SigAddressPrepareStage1) -- 21786 signatures processed. 66 are IP-only rules, 6601 are inspecting packet payload, 15001 inspect application layer, 0 are decoder event only [8171] 27/5/2019 -- 02:02:34 - (util-runmodes.c:297) (RunModeSetLiveCaptureWorkersForDevice) -- Going to use 4 thread(s) [8199] 27/5/2019 -- 02:02:34 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [8199] 27/5/2019 -- 02:02:34 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 19 files. [8200] 27/5/2019 -- 02:02:34 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [8200] 27/5/2019 -- 02:02:34 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 19 files. [8201] 27/5/2019 -- 02:02:34 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [8201] 27/5/2019 -- 02:02:34 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 19 files. [8202] 27/5/2019 -- 02:02:34 - (log-pcap.c:762) (PcapLogInitRingBuffer) -- Initializing PCAP ring buffer for /data/nsm//log.%n.%t.pcap. [8202] 27/5/2019 -- 02:02:34 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 19 files. [8171] 27/5/2019 -- 02:02:34 - (util-conf.c:115) (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket [8171] 27/5/2019 -- 02:02:34 - (unix-manager.c:131) (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket' [8171] 27/5/2019 -- 02:02:34 - (tm-threads.c:2157) (TmThreadWaitOnThreadInit) -- all 4 packet processing threads, 4 management threads initialized, engine started. [8202] 27/5/2019 -- 02:02:34 - (source-af-packet.c:509) (AFPPeersListReachedInc) -- All AFP capture threads are running.

selks-user@SELKS:/var/log/suricata$

pevma commented 5 years ago

ok seems better. what is the timestamp output of the logs by the command:

grep '"event_type": "alert"' /var/log/suricata/eve-alert.json | tail -1
Mauro2k commented 5 years ago

No data..

image

pevma commented 5 years ago

The command dies not appear to be the same as I quoted in my previous response - seems there is a space missing in your execution ?

-- Regards, Peter Manev

On 27 May 2019, at 17:04, Mac notifications@github.com wrote:

No data..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Mauro2k commented 5 years ago

The file eve-alert.json does not exist. There is the file eve.json. Is the same?

image

pevma commented 5 years ago

Yes - sorry - eve.json

So it means there are no alerts generated - and having in mind Suricata seems to be running there could be something else that changed ?

-- Regards, Peter Manev

On 27 May 2019, at 18:40, Mac notifications@github.com wrote:

The file eve-alert.json does not exist. There is the file eve.json. Is the same?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Mauro2k commented 5 years ago

I have not changed anything. It is the initial installation. I can not find the problem. I'm disoriented.

pevma commented 5 years ago

If you have not changed anything - it means that maybe something changed on the mirror/traffic port ?

Mauro2k commented 5 years ago

I've thought about that. But if it was a problem in the PortMirroring configuration, why do I see the packages in the EVEBOX?

Mauro2k commented 5 years ago

I remind you that this installation is made on an ESXi.

pevma commented 5 years ago

Do you see Alerts in Evebox where you do not see them for that amount of time in Scirius/Kibana ?

Mauro2k commented 5 years ago

No, I only see traffic but not alerts!

pevma commented 5 years ago

ok - so all apps do not see alerts as there are none in the DB (ES) so in that case the change must be elsewhere, somewhere on the host or something changed in the traffic. But i myself am not sure what exactly could that be. If you do tcpdump do you see something abnormal in the traffic capture (for example via wireshark if the size of the pcap is not that huge)?

Mauro2k commented 5 years ago

I spent some time analyzing the traffic with tcpdump and I do not see anything strange. The sniffing interface receives traffic from all the vlans.

pevma commented 5 years ago

Any offloading or warnings observed by wireshark ? Do you have vlan tracking enabled in /etc/suricata/selks5-addins.yaml ? ( https://github.com/OISF/suricata/blob/master/suricata.yaml.in#L1229 )

Mauro2k commented 5 years ago

Yes, i Have it. image

Flow-timeouts: by default

image

Mauro2k commented 5 years ago

I just checked the fast.log file and I see that it has records. When reviewing it, I notice that there are some alarms since 25/5 but only appears in the DASHBOARD the last one detected.

Fast.log image

Dashboard: image

At the same time I have performed the following test and it does not appear:

curl -A "BlackSun" www.google.com

Before the reinstallation I did it.

This is very strange, I do not understand what is happening

pevma commented 5 years ago

The records above appear to be from 25 May and 30 May. (Also just for info SELKS does not utilize fast.log as it is legacy. ) So those same records should be present in /var/log/suricata/eve.json Could you try this - (i realized there was an extra space befor "alert" in the command before)

grep '"event_type":"alert"' /var/log/suricata/eve.json | tail -5 | jq .
pevma commented 5 years ago

Also check the timezone set up onn the box /cmd (date) and in Scirius (Settings , from the drop down menu in the upper left corner)

Mauro2k commented 5 years ago

result of "grep '"event_type":"alert"' /var/log/suricata/eve.json | tail -5 | jq ."

selks-user@SELKS:/var/log/suricata$ grep '"event_type":"alert"' /var/log/suricata/eve.json | tail -5 | jq . { "timestamp": "2019-05-31T10:14:19.582621-0300", "flow_id": 119216089588701, "in_iface": "ens192", "event_type": "alert", "src_ip": "192.168.21.132", "src_port": 57621, "dest_ip": "192.168.21.255", "dest_port": 57621, "proto": "UDP", "alert": { "action": "allowed", "gid": 1, "signature_id": 2027397, "rev": 1, "signature": "ET POLICY Spotify P2P Client", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "updated_at": [ "2019_05_30" ], "performance_impact": [ "Low" ], "created_at": [ "2019_05_30" ], "signature_severity": [ "Minor" ], "deployment": [ "Internal" ], "attack_target": [ "Client_Endpoint" ], "affected_product": [ "Windows_Client_Apps" ] } }, "app_proto": "failed", "flow": { "pkts_toserver": 1, "pkts_toclient": 0, "bytes_toserver": 86, "bytes_toclient": 0, "start": "2019-05-31T10:14:19.582621-0300" }, "payload": "U3BvdFVkcDBLCKgSAE27EAABAARIlcIDo3coC8mysGtyrPsNoy5gZZI4DZY=", "payload_printable": "SpotUdp0K....M......H....w(....kr..\r..e.8\r.", "stream": 0, "packet": "////////aPcoseVRCABFAABIuJFAAEAR1T/AqBWEwKgV/+EV4RUANPg3U3BvdFVkcDBLCKgSAE27EAABAARIlcIDo3coC8mysGtyrPsNoy5gZZI4DZY=", "packet_info": { "linktype": 1 } } { "timestamp": "2019-05-31T10:18:10.354490-0300", "flow_id": 2149754235676858, "in_iface": "ens192", "event_type": "alert", "src_ip": "192.168.21.137", "src_port": 57621, "dest_ip": "192.168.21.255", "dest_port": 57621, "proto": "UDP", "alert": { "action": "allowed", "gid": 1, "signature_id": 2027397, "rev": 1, "signature": "ET POLICY Spotify P2P Client", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "updated_at": [ "2019_05_30" ], "performance_impact": [ "Low" ], "created_at": [ "2019_05_30" ], "signature_severity": [ "Minor" ], "deployment": [ "Internal" ], "attack_target": [ "Client_Endpoint" ], "affected_product": [ "Windows_Client_Apps" ] } }, "app_proto": "failed", "flow": { "pkts_toserver": 1, "pkts_toclient": 0, "bytes_toserver": 86, "bytes_toclient": 0, "start": "2019-05-31T10:18:10.354490-0300" }, "payload": "U3BvdFVkcDD3W4ozF+ltNAABAARIlcIDNICyxlv9NPurvFhNklmUWDhYuGE=", "payload_printable": "SpotUdp0.[.3..m4....H...4...[.4...XM.Y.X8X.a", "stream": 0, "packet": "////////UHudawebCABFAABILpsAAIARXzHAqBWJwKgV/+EV4RUANGMHU3BvdFVkcDD3W4ozF+ltNAABAARIlcIDNICyxlv9NPurvFhNklmUWDhYuGE=", "packet_info": { "linktype": 1 } } { "timestamp": "2019-05-31T10:18:08.840777-0300", "flow_id": 1644968876758089, "in_iface": "ens192", "event_type": "alert", "src_ip": "192.168.21.33", "src_port": 57621, "dest_ip": "192.168.21.255", "dest_port": 57621, "proto": "UDP", "alert": { "action": "allowed", "gid": 1, "signature_id": 2027397, "rev": 1, "signature": "ET POLICY Spotify P2P Client", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "updated_at": [ "2019_05_30" ], "performance_impact": [ "Low" ], "created_at": [ "2019_05_30" ], "signature_severity": [ "Minor" ], "deployment": [ "Internal" ], "attack_target": [ "Client_Endpoint" ], "affected_product": [ "Windows_Client_Apps" ] } }, "app_proto": "failed", "flow": { "pkts_toserver": 1, "pkts_toclient": 0, "bytes_toserver": 86, "bytes_toclient": 0, "start": "2019-05-31T10:18:08.840777-0300" }, "payload": "U3BvdFVkcDA+sSRJnnVEtgABAARIlcIDgxZH3wMxiDh2y9i/47lonyA9QCM=", "payload_printable": "SpotUdp0>.$I.uD.....H.....G..1.8v.....h. =@#", "stream": 0, "packet": "////////BNOwwWJKCABFAABI0hdAAEARvBzAqBUhwKgV/+EV4RUANGUHU3BvdFVkcDA+sSRJnnVEtgABAARIlcIDgxZH3wMxiDh2y9i/47lonyA9QCM=", "packet_info": { "linktype": 1 } } { "timestamp": "2019-05-31T10:18:16.343879-0300", "flow_id": 476765690019655, "in_iface": "ens192", "event_type": "alert", "src_ip": "192.168.7.116", "src_port": 57621, "dest_ip": "192.168.7.255", "dest_port": 57621, "proto": "UDP", "alert": { "action": "allowed", "gid": 1, "signature_id": 2027397, "rev": 1, "signature": "ET POLICY Spotify P2P Client", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "updated_at": [ "2019_05_30" ], "performance_impact": [ "Low" ], "created_at": [ "2019_05_30" ], "signature_severity": [ "Minor" ], "deployment": [ "Internal" ], "attack_target": [ "Client_Endpoint" ], "affected_product": [ "Windows_Client_Apps" ] } }, "app_proto": "failed", "flow": { "pkts_toserver": 1, "pkts_toclient": 0, "bytes_toserver": 86, "bytes_toclient": 0, "start": "2019-05-31T10:18:16.343879-0300" }, "payload": "U3BvdFVkcDB80YrwoXSOTQABAARIlcIDwCdMBYKAIrYGahYbFqxWT9BhF5I=", "payload_printable": "SpotUdp0|....t.M....H....'L...\"..j....VO.a..", "stream": 0, "packet": "////////jBZFiQcXCABFAABInahAAEARDDnAqAd0wKgH/+EV4RUANL8iU3BvdFVkcDB80YrwoXSOTQABAARIlcIDwCdMBYKAIrYGahYbFqxWT9BhF5I=", "packet_info": { "linktype": 1 } } { "timestamp": "2019-05-31T10:18:40.125923-0300", "flow_id": 615731505982435, "in_iface": "ens192", "event_type": "alert", "src_ip": "192.168.21.132", "src_port": 57621, "dest_ip": "192.168.21.255", "dest_port": 57621, "proto": "UDP", "alert": { "action": "allowed", "gid": 1, "signature_id": 2027397, "rev": 1, "signature": "ET POLICY Spotify P2P Client", "category": "Not Suspicious Traffic", "severity": 3, "metadata": { "updated_at": [ "2019_05_30" ], "performance_impact": [ "Low" ], "created_at": [ "2019_05_30" ], "signature_severity": [ "Minor" ], "deployment": [ "Internal" ], "attack_target": [ "Client_Endpoint" ], "affected_product": [ "Windows_Client_Apps" ] } }, "app_proto": "failed", "flow": { "pkts_toserver": 1, "pkts_toclient": 0, "bytes_toserver": 86, "bytes_toclient": 0, "start": "2019-05-31T10:18:40.125923-0300" }, "payload": "U3BvdFVkcDBLCKgSAE27EAABAARIlcIDo3coC8mysGtyrPsNoy5gZZI4DZY=", "payload_printable": "SpotUdp0K....M......H....w(....kr..\r..e.8\r.", "stream": 0, "packet": "////////aPcoseVRCABFAABI43VAAEARqlvAqBWEwKgV/+EV4RUANPg3U3BvdFVkcDBLCKgSAE27EAABAARIlcIDo3coC8mysGtyrPsNoy5gZZI4DZY=", "packet_info": { "linktype": 1 } } selks-user@SELKS:/var/log/suricata$

image

I see that it started to detect one of the new recently added rules of Spotify, but I still do not see others that I generate on my own (Torrents, Blacksun, malware)

pevma commented 5 years ago

Ok- so the logging is functional/up and running. Now you we need to concentrate on the case that it does not alert and investigate why. Case by case to start with would be preferable I suggest.