StamusNetworks / SELKS

A Suricata based IDS/IPS/NSM distro
https://www.stamus-networks.com/open-source/#selks
GNU General Public License v3.0
1.48k stars 285 forks source link

import pcap #192

Closed sienikam closed 5 years ago

sienikam commented 5 years ago

Hello,

is there a way to import pcap file (from tcpdump) into SELKS to have full packet analyse etc?

pevma commented 5 years ago

yes sure! you would need to do something like -

suricata -k none -r /path/to/pcap --runmode=autopfp

Then you can look at the dashboards and similar with the appropriate timespan (from the pcap)

sienikam commented 5 years ago

root@SELKS:/home/selks-user# suricata -k none -r dump.pcap --runmode=autopfp [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'rule-files' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'classification-file' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'detect' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'stats' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'outputs' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'logging' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'app-layer' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'af-packet' redefined. [28977] 4/8/2019 -- 18:01:26 - (suricata.c:1071) (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [28977] 4/8/2019 -- 18:01:26 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [28977] 4/8/2019 -- 18:01:26 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [28977] 4/8/2019 -- 18:01:53 - (runmodes.c:359) (RunModeDispatch) -- [ERRCODE: SC_ERR_RUNMODE(187)] - The custom type "autopfp" doesn't exist for this runmode type "PCAP_FILE". Please use --list-runmodes to see available custom types for this runmode

root@SELKS:/home/selks-user# suricata -V This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19)

any idea what is wrong?

pevma commented 5 years ago

Hi, Try passing “—runmode=autofp” To the command line.

Thank you

-- Regards, Peter Manev

On 4 Aug 2019, at 19:25, Kamil Sienicki notifications@github.com wrote:

root@SELKS:/home/selks-user# suricata -k none -r dump.pcap --runmode=autopfp [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'rule-files' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'classification-file' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'detect' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'stats' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'outputs' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'logging' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'app-layer' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [28977] 4/8/2019 -- 18:01:26 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'af-packet' redefined. [28977] 4/8/2019 -- 18:01:26 - (suricata.c:1071) (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [28977] 4/8/2019 -- 18:01:26 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [28977] 4/8/2019 -- 18:01:26 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [28977] 4/8/2019 -- 18:01:53 - (runmodes.c:359) (RunModeDispatch) -- [ERRCODE: SC_ERR_RUNMODE(187)] - The custom type "autopfp" doesn't exist for this runmode type "PCAP_FILE". Please use --list-runmodes to see available custom types for this runmode

root@SELKS:/home/selks-user# suricata -V This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19)

any idea what is wrong?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sienikam commented 5 years ago

Now it looks better:

/home/selks-user# suricata -k none -r dump.pcap --runmode=autofp [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'rule-files' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'classification-file' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'detect' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'stats' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'outputs' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'logging' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'app-layer' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'af-packet' redefined. [2357] 4/8/2019 -- 20:10:17 - (suricata.c:1071) (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [2357] 4/8/2019 -- 20:10:17 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [2357] 4/8/2019 -- 20:10:17 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [2363] 4/8/2019 -- 20:10:45 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 2 files. [2364] 4/8/2019 -- 20:10:45 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 3 files. [2357] 4/8/2019 -- 20:10:45 - (tm-threads.c:2145) (TmThreadWaitOnThreadInit) -- all 3 packet processing threads, 4 management threads initialized, engine started. [2357] 4/8/2019 -- 20:10:45 - (suricata.c:2851) (SuricataMainLoop) -- Signal Received. Stopping engine. [2362] 4/8/2019 -- 20:10:45 - (source-pcap-file.c:378) (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 1233 packets, 223147 bytes

but anyway there is no data or alerts shown in evebox, kibana or scirius.. I have tried to restart suricata etc but didn't help...

pevma commented 5 years ago

Tha data should follow the timestamps from the pcap. Maybe you need to adjust the timespan in the dashboards ?

-- Regards, Peter Manev

On 4 Aug 2019, at 21:12, Kamil Sienicki notifications@github.com wrote:

Now it looks better:

/home/selks-user# suricata -k none -r dump.pcap --runmode=autofp [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'rule-files' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'classification-file' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'detect' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'stats' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'outputs' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'logging' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'app-layer' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [2357] 4/8/2019 -- 20:10:17 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'af-packet' redefined. [2357] 4/8/2019 -- 20:10:17 - (suricata.c:1071) (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [2357] 4/8/2019 -- 20:10:17 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [2357] 4/8/2019 -- 20:10:17 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [2363] 4/8/2019 -- 20:10:45 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 2 files. [2364] 4/8/2019 -- 20:10:45 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 3 files. [2357] 4/8/2019 -- 20:10:45 - (tm-threads.c:2145) (TmThreadWaitOnThreadInit) -- all 3 packet processing threads, 4 management threads initialized, engine started. [2357] 4/8/2019 -- 20:10:45 - (suricata.c:2851) (SuricataMainLoop) -- Signal Received. Stopping engine. [2362] 4/8/2019 -- 20:10:45 - (source-pcap-file.c:378) (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 1233 packets, 223147 bytes

but anyway there is no data or alerts shown in evebox, kibana or scirius..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sienikam commented 5 years ago

I have timestamp always like 24h and there is no data from pcap visible at all. What about these warning from suricata import?

2357] 4/8/2019 -- 20:10:17 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [2357] 4/8/2019 -- 20:10:17 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [2363] 4/8/2019 -- 20:10:45 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 2 files.

Is there any need to change something in configuration?

pevma commented 5 years ago

You can disregard those warning for the moment. What are the timestamps in the pcap (not the dashboard) ?

sienikam commented 5 years ago

image

an example of HTTP request to http://testmyids.com

for example - after importing this pcap to suricata alert is not shown in dasboard.

dump.pcap file - https://ufile.io/m1n0kqz4

in addition - when I try curl testmyids.com at localhost SELKS machine then alert trigger immediately

pevma commented 5 years ago

Ok thanks Do you see the specific http requests in the SN-HTTP dashboard ?

-- Regards, Peter Manev

On 5 Aug 2019, at 13:36, Kamil Sienicki notifications@github.com wrote:

an example of HTTP request to http://testmyids.com

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sienikam commented 5 years ago

also not.

pevma commented 5 years ago

That would probably be an indication that Suricata is not seeing the traffic. Is there anything at all related to that (testmyids) in /var/log/suricata/eve.json ?

-- Regards, Peter Manev

On 5 Aug 2019, at 15:37, Kamil Sienicki notifications@github.com wrote:

also not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sienikam commented 5 years ago

a little bit hard for me to search in that file.. but from what I can check there is no data related to testmyids..

during suricata import I have found following entry in verbose mode: [45152] 6/8/2019 -- 09:42:17 - (counters.c:841) (StatsLogSummary) -- Alerts: 2

pevma commented 5 years ago

If Suri sees the traffic i was thinking you should have something related. Does the following command -

grep -i "testmyids" /var/log/suricata/eve.json

return a result?

sienikam commented 5 years ago

none result..

pevma commented 5 years ago

It would mean most likely the traffic is not mirrored properly (?) or Suricata is not sniffing on the right interface maybe?

-- Regards, Peter Manev

On 6 Aug 2019, at 11:25, Kamil Sienicki notifications@github.com wrote:

no result..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ph1sec commented 5 years ago

Hi pevma and k3s4, dont want to hijack this thread, but i think my setup can reproduce this issue as well. Ive updated the whole SELKS install to match the output of k3s4 import. Further ive setup a plain Win10 machine and ran Nextrons APT sim to generate some noise for suri (https://github.com/NextronSystems/APTSimulator). Traffic was captured via tcpdump on the Win10 Host and then imported via the discussed command.

/home/selks-user# suricata -k none -r aptsim_00001_20190805153601.pcap --runmode=autofp [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:255) <Info> (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'rule-files' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'classification-file' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'detect' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'stats' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'outputs' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'logging' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'app-layer' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:255) <Info> (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) <Info> (ConfYamlParse) -- Configuration node 'af-packet' redefined. [5974] 6/8/2019 -- 03:52:04 - (suricata.c:1071) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [5974] 6/8/2019 -- 03:52:04 - (counters.c:264) <Warning> (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [5974] 6/8/2019 -- 03:52:04 - (output-json-stats.c:467) <Warning> (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [5978] 6/8/2019 -- 03:52:22 - (log-pcap.c:903) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 1 files. [5979] 6/8/2019 -- 03:52:22 - (log-pcap.c:903) <Notice> (PcapLogInitRingBuffer) -- Ring buffer initialized with 1 files. [5974] 6/8/2019 -- 03:52:22 - (tm-threads.c:2145) <Notice> (TmThreadWaitOnThreadInit) -- all 3 packet processing threads, 4 management threads initialized, engine started. [5974] 6/8/2019 -- 03:52:22 - (suricata.c:2851) <Notice> (SuricataMainLoop) -- Signal Received. Stopping engine. [5977] 6/8/2019 -- 03:52:22 - (source-pcap-file.c:378) <Notice> (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 1601 packets, 441216 bytes

Eve.json does not contain any of the 1601 imported packets (which are not corrupt according to a quick Wireshark review). To clarify, the pcap is not from the SELKS probe but from another host (captured via tcpdump including full sized packets). Could this be an issue?

Cheers Phil

pevma commented 5 years ago

No issue , please feel free to report ! :) You need to specify a log folder on the command line. Aka add to you command line “-l /var/log/suricata/ “

Would that make any diff? Also check/tail - start - tail -F /var/log/suricata/eve.json before the run , then do the run and see if events are added to the eve.json

Thanks

-- Regards, Peter Manev

On 6 Aug 2019, at 12:34, ph1sec notifications@github.com wrote:

Hi pevma and k3s4, dont want to hijack this thread, but i think my setup can reproduce this issue as well. Ive updated the whole SELKS install to match the output of k3s4 import. Further ive setup a plain Win10 machine and ran Nextrons APT sim to generate some noise for suri (https://github.com/NextronSystems/APTSimulator). Traffic was captured via tcpdump on the Win10 Host and then imported via the discussed command.

/home/selks-user# suricata -k none -r aptsim_00001_20190805153601.pcap --runmode=autofp [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-addin.yaml. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-rule-path' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'rule-files' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'classification-file' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'reference-config-file' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'detect' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'default-log-dir' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'stats' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'outputs' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'logging' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'app-layer' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'asn1-max-frames' redefined. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:255) (ConfYamlParse) -- Including configuration file /etc/suricata/selks5-interfaces-config.yaml. [5974] 6/8/2019 -- 03:52:04 - (conf-yaml-loader.c:279) (ConfYamlParse) -- Configuration node 'af-packet' redefined. [5974] 6/8/2019 -- 03:52:04 - (suricata.c:1071) (LogVersion) -- This is Suricata version 5.0.0-dev (06d3e1d3d 2019-07-19) running in USER mode [5974] 6/8/2019 -- 03:52:04 - (counters.c:264) (StatsInitCtxPreOutput) -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder..' to 'decoder.event..'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml. [5974] 6/8/2019 -- 03:52:04 - (output-json-stats.c:467) (OutputStatsLogInitSub) -- [ERRCODE: SC_WARN_EVE_MISSING_EVENTS(318)] - eve.stats will not display all decoder events correctly. See #2225. Set a prefix in stats.decoder-events-prefix. In 5.0 the prefix will default to 'decoder.event'. [5978] 6/8/2019 -- 03:52:22 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 1 files. [5979] 6/8/2019 -- 03:52:22 - (log-pcap.c:903) (PcapLogInitRingBuffer) -- Ring buffer initialized with 1 files. [5974] 6/8/2019 -- 03:52:22 - (tm-threads.c:2145) (TmThreadWaitOnThreadInit) -- all 3 packet processing threads, 4 management threads initialized, engine started. [5974] 6/8/2019 -- 03:52:22 - (suricata.c:2851) (SuricataMainLoop) -- Signal Received. Stopping engine. [5977] 6/8/2019 -- 03:52:22 - (source-pcap-file.c:378) (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 1601 packets, 441216 bytes

Eve.json does not contain any of the 1601 imported packets (which are not corrupt according to a quick Wireshark review). To clarify, the pcap is not from the SELKS probe but from another host (captured via tcpdump including full sized packets). Could this be an issue?

Cheers Phil

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ph1sec commented 5 years ago

Hi Pevma,

'-l /var/log/suricata/'

Makes a difference, now i see the imported events within evebox. Will check if everything is there and report back.

sienikam commented 5 years ago

also works fine for me with '-l /var/log/suricata/'

full command should be: suricata -k none -r dump.pcap --runmode=autofp -l /var/log/suricata/

thanks for that :)

ph1sec commented 5 years ago

Also confirmed from my end, tested another pcap, everything is now properly processed within SELKS.

Thanks for the fast response Pevma!

pevma commented 5 years ago

Ok cool. Thanks for the feedback - please keep us posted if anything. (Not sure if I noticed before - you could also add “-k none” to the command line too)

Related to - https://redmine.openinfosecfoundation.org/issues/3095

On 6 Aug 2019, at 13:16, ph1sec notifications@github.com wrote:

Also confirmed from my end, tested another pcap, everything is now properly processed within SELKS.

Thanks for the fast response Pevma!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

pevma commented 5 years ago

Glad to hear Kamil ! Sorry I missed that earlier.

-- Regards, Peter Manev

On 6 Aug 2019, at 13:08, Kamil Sienicki notifications@github.com wrote:

also works fine for me with '-l /var/log/suricata/'

full command should be: suricata -k none -r dump.pcap --runmode=autofp -l /var/log/suricata/

thanks for that :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sienikam commented 5 years ago

one more question :) is there a way that suricata will read pcap file continuously ? I have tcpdump running and want to read .pcap file and import data to suricata on fly - is there such option? I have found option "--pcap-file-continuous" in suricata but it give me following error

[28301] 7/8/2019 -- 14:07:23 - (source-pcap-file-helper.c:142) (PcapFileDispatch) -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -1 truncated dump file; tried to read 64 captured bytes, only got 51 for dump.pcap

pevma commented 5 years ago

Yes you need to point it to a folder. And it will continue to read from that folder til the it went through all pcaps or the folder is empty.

sienikam commented 5 years ago

pevma - thanks for that :) I've already setup suricata continous pcap reading and it works fine :) I have only one strange issue which is not quite clear for me.. after visiting webpage like - testmyids.com source IP address in evebox and other similar tools always appear as - 192.168.1.1 (router ip) and destination address is always SELKS machine no matter from which machine in my LAN network I have accessed that page - this is how it looks like now:

image

so source ip address and destination is wrong and I have no idea why.. I have checked pcap file in wireshark and source ip and destination is correct:

image

any idea what can be wrong? is this something related to suricata pcap import?

pevma commented 5 years ago

How did you generate that pcap? Was it from the same sniffing interface of SELKS? (if not that can explain the difference)

sienikam commented 5 years ago

the problem was because there was running another suricata daemon in background and I was running suricata import pcap at the same time - that's why there was some mess in evebox ;) anyway thanks for help as now I have my IDS/IPS working correctly :)

pevma commented 5 years ago

Thank you for the follow up! Glad it is solved!

nimaforoughi commented 2 years ago

[8835] 1/1/2022 -- 21:02:10 - (tm-threads.c:1888) (TmThreadWaitOnThreadInit) -- all 5 packet processing threads, 4 management threads initialized, engine started. [8835] 1/1/2022 -- 21:02:11 - (suricata.c:2602) (SuricataMainLoop) -- Signal Received. Stopping engine. [8838] 1/1/2022 -- 21:02:12 - (source-pcap-file.c:376) (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 14841 packets, 8406713 bytes suricata: host.c:314: HostShutdown: Assertion `!(extension ({ auto_type __atomic_load_ptr = (&(h->use_cnt_sc_atomic)); typeof (*atomic_load_ptr) atomic_load_tmp; __atomic_load (atomic_load_ptr, &atomic_load_tmp, (5)); __atomic_load_tmp; }) > 0)' failed. Aborted

I face this problem and it does not ingest anything. Any Idea?

pevma commented 2 years ago

What is the command you execute ?

-- Regards, Peter Manev

On 2 Jan 2022, at 03:09, nimaforoughi @.***> wrote:

 [8835] 1/1/2022 -- 21:02:10 - (tm-threads.c:1888) (TmThreadWaitOnThreadInit) -- all 5 packet processing threads, 4 management threads initialized, engine started. [8835] 1/1/2022 -- 21:02:11 - (suricata.c:2602) (SuricataMainLoop) -- Signal Received. Stopping engine. [8838] 1/1/2022 -- 21:02:12 - (source-pcap-file.c:376) (ReceivePcapFileThreadExitStats) -- Pcap-file module read 1 files, 14841 packets, 8406713 bytes suricata: host.c:314: HostShutdown: Assertion `!(extension ({ auto_type atomic_load_ptr = (&(h->use_cnt_sc_atomic)); typeof (*atomic_load_ptr) atomic_load_tmp; atomic_load (atomic_load_ptr, &atomic_load_tmp, (5)); __atomic_load_tmp; }) > 0)' failed. Aborted

I face this problem and it does not ingest anything. Any Idea?

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you modified the open/close state.