Open whale-withme opened 1 month ago
Not in pcap file, I tried interface and faced the same problem. Should I ignore this?
Hi @whale-withme, please always report the versions of slips, commit and system you are using. If not it is impossible to know. You can also try -e 3 to have more debugging messages.
Hi! @eldraco I use slips version 1.0.15
and in master branch.
Did it meet some data format conversion problem? I guess
Oh, my system is in docker. Sorry, I'm new.
Can you provide the error.log file that slips generates? and maybe the whole slips.log too? it is usually in the output folder. In your case it would be something like output/test7-malicious.pcap/slips.log
and similar for error.log
Sure! Here is error.log
2024/07/17 09:26:32.985524 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:32.985672 [Flow ML Detection] Error in train()
2024/07/17 09:26:32.985774 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:34.204108 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:34.204286 [Flow ML Detection] Error in train()
2024/07/17 09:26:34.204408 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:34.960821 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:34.960999 [Flow ML Detection] Error in train()
2024/07/17 09:26:34.961105 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:35.729694 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:35.729911 [Flow ML Detection] Error in train()
2024/07/17 09:26:35.730036 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:36.536441 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:36.536671 [Flow ML Detection] Error in train()
2024/07/17 09:26:36.536795 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:37.371597 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:37.371922 [Flow ML Detection] Error in train()
2024/07/17 09:26:37.372059 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:38.273439 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
dataset = dataset.drop(field, axis=1)
File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
return super().drop(
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
obj = obj._drop_axis(labels, axis, level=level, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
new_axis = axis.drop(labels, errors=errors)
File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"
2024/07/17 09:26:38.273839 [Flow ML Detection] Error in train()
2024/07/17 09:26:38.273994 [Flow ML Detection] Traceback (most recent call last):
File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'
2024/07/17 09:26:52.434125 [Update Manager] Connection error while downloading the file https://check.torproject.org/torbulkexitlist. Aborting.
Here is slips.log
2024/07/17 09:26:25.390494 [Main] Using redis server on port: [32m6379[0m
2024/07/17 09:26:25.390760 [Main] Started [32mMain[0m process [PID [32m3511[0m]
2024/07/17 09:26:25.395186 [Main] Started [32mPBar[0m process [PID [32m3521[0m]
2024/07/17 09:26:25.474654 [Main] Starting modules
2024/07/17 09:26:27.235406 [Main] Starting the module [32mARP[0m (Detect ARP attacks) [PID [32m3619[0m]
2024/07/17 09:26:27.255412 [Main] Starting the module [32mFlow Alerts[0m (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID [32m3622[0m]
2024/07/17 09:26:27.265072 [Main] Starting the module [32mFlow ML Detection[0m (Train or test a Machine Learning model to detect malicious flows) [PID [32m3623[0m]
2024/07/17 09:26:27.275282 [Main] Starting the module [32mHTTP Analyzer[0m (Analyze HTTP flows) [PID [32m3624[0m]
2024/07/17 09:26:27.284780 [Main] Starting the module [32mIP Info[0m (Get different info about an IP/MAC address) [PID [32m3625[0m]
2024/07/17 09:26:27.293551 [Main] Starting the module [32mNetwork Discovery[0m (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID [32m3626[0m]
2024/07/17 09:26:27.303480 [Main] Starting the module [32mRisk IQ[0m (Module to get passive DNS info about IPs from RiskIQ) [PID [32m3627[0m]
2024/07/17 09:26:27.312780 [Main] Starting the module [32mRNN C&C Detection[0m (Detect C&C channels based on behavioral letters) [PID [32m3628[0m]
2024/07/17 09:26:27.345828 [Main] Starting the module [32mThreat Intelligence[0m (Check if the source IP or destination IP are in a malicious list of IPs) [PID [32m3726[0m]
2024/07/17 09:26:27.356497 [Main] Starting the module [32mTimeline[0m (Creates kalipso timeline of what happened in the network based on flows and available data) [PID [32m3728[0m]
2024/07/17 09:26:27.370486 [Main] Starting the module [32mUpdate Manager[0m (Update Threat Intelligence files) [PID [32m3729[0m]
2024/07/17 09:26:27.378817 [Update Manager] Checking if we need to download TI files.
2024/07/17 09:26:27.381869 [Main] Starting the module [32mVirustotal[0m (IP, domain and file hash lookup on Virustotal) [PID [32m3730[0m]
2024/07/17 09:26:27.882654 [Main] Disabled Modules: ['template', 'ensembling', 'exporting_alerts', 'p2ptrust', 'cesnet', 'blocking', 'leak_detector', 'cyst']
2024/07/17 09:26:27.890380 [Evidence] Storing Slips logs in output/test7-malicious.pcap_2024-07-17_09:26:23/
2024/07/17 09:26:27.897297 [Main] Started [32mEvidence Process[0m [PID [32m3731[0m]
2024/07/17 09:26:27.909913 [Main] Started [32mProfiler Process[0m [PID [32m3732[0m]
2024/07/17 09:26:27.912159 [Main] Metadata added to output/test7-malicious.pcap_2024-07-17_09:26:23/metadata
2024/07/17 09:26:27.921421 [Main] Started [32mInput Process[0m [PID [32m3733[0m]
2024/07/17 09:26:27.924313 [Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
2024/07/17 09:26:27.931409 [Input] Storing zeek log files in output/test7-malicious.pcap_2024-07-17_09:26:23/zeek_files
2024/07/17 09:26:32.962922 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 51.0.
2024/07/17 09:26:32.981987 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:34.189028 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 101.0.
2024/07/17 09:26:34.203826 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:34.943573 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 151.0.
2024/07/17 09:26:34.960564 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:35.712896 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 201.0.
2024/07/17 09:26:35.729430 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:36.517875 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 251.0.
2024/07/17 09:26:36.536187 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:37.351208 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 301.0.
2024/07/17 09:26:37.371325 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:38.208675 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 351.0.
2024/07/17 09:26:38.273124 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:52.436421 [Update Manager] 43 TI files successfully loaded.
2024/07/17 09:27:00.941889 [Input] We read everything. No more input. Stopping input process. Sent 541 lines
2024/07/17 09:27:00.941984 [Input] Telling Profiler to stop because no more input is arriving.
2024/07/17 09:27:00.942044 [Input] Waiting for Profiler to stop.
2024/07/17 09:27:00.942222 [Profiler] Stopping. Total lines read: 541
2024/07/17 09:27:00.942342 [Profiler] Marking Profiler as done processing.
2024/07/17 09:27:00.942378 [Profiler] Profiler is done processing.
2024/07/17 09:27:00.942466 [Profiler] Profiler is done telling input.py that it's done processing.
2024/07/17 09:27:00.942499 [Profiler] Stopping. Total lines read: 541
2024/07/17 09:27:00.942485 [Input] Input is done processing.
2024/07/17 09:27:01.378260 [Input] Stopping. Total lines read: 541
2024/07/17 09:27:03.016316 [Main]
---------------------------
2024/07/17 09:27:03.016557 [Main] Stopping Slips
2024/07/17 09:27:03.034812 [Main] Analysis of dataset/test7-malicious.pcap finished in 0.63 minutes
2024/07/17 09:27:03.035100 [Main] Total flows read (without altflows): 374
2024/07/17 09:27:03.035815 [Main] [32mUpdate Manager [0m Stopped. [32m13[0m left.
2024/07/17 09:27:03.035957 [Main] [32mProfiler [0m Stopped. [32m12[0m left.
2024/07/17 09:27:03.049347 [Main] [32mTimeline [0m Stopped. [32m11[0m left.
2024/07/17 09:27:03.342716 [Main] [32mIP Info [0m Stopped. [32m10[0m left.
2024/07/17 09:27:06.345985 [Main] [32mRNN C&C Detection [0m Stopped. [32m9[0m left.
2024/07/17 09:27:06.346397 [Main] [32mThreat Intelligence [0m Stopped. [32m8[0m left.
2024/07/17 09:27:06.346725 [Main] [32mARP [0m Stopped. [32m7[0m left.
2024/07/17 09:27:06.347016 [Main] [32mFlow ML Detection [0m Stopped. [32m6[0m left.
2024/07/17 09:27:06.347293 [Main] [32mHTTP Analyzer [0m Stopped. [32m5[0m left.
2024/07/17 09:27:06.347548 [Main] [32mNetwork Discovery [0m Stopped. [32m4[0m left.
2024/07/17 09:27:06.347819 [Main] [32mInput [0m Stopped. [32m3[0m left.
2024/07/17 09:27:06.348077 [Main] [32mProgress Bar [0m Stopped. [32m2[0m left.
2024/07/17 09:27:06.348283 [Main] The following modules are busy working on your data.
['Flow Alerts', 'Evidence']
You can wait for them to finish, or you can press CTRL-C again to force-kill.
2024/07/17 09:27:19.942328 [Main] [32mEvidence [0m Stopped. [32m1[0m left.
2024/07/17 09:27:19.964624 [Main] [32mFlow Alerts [0m Stopped. [32m0[0m left.
2024/07/17 09:27:19.968192 [Main] [Process Manager] Slips didn't shutdown gracefully - User pressed ctr+c or Slips was killed by the OS
In Linux with Docker, for us it works without errors.
Linux 3c0d724d10ba 5.15.0-75-generic #82~20.04.1-Ubuntu
Docker version 27.0.3, build 7d4bcd8
./slips.py -f dataset/test7-malicious.pcap -c config/slips.conf -e 1
https://stratosphereips.org
---------------------------
[Main] Using redis server on port: 6379
Started Main process [PID 31]
Started PBar process [PID 48]
Starting modules
Starting the module ARP (Detect ARP attacks) [PID 74]
Starting the module Flow Alerts (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID 77]
Starting the module Flow ML Detection (Train or test a Machine Learning model to detect malicious flows) [PID 78]
Starting the module HTTP Analyzer (Analyze HTTP flows) [PID 79]
Starting the module IP Info (Get different info about an IP/MAC address) [PID 80]
Starting the module Network Discovery (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID 81]
Starting the module Risk IQ (Module to get passive DNS info about IPs from RiskIQ) [PID 82]
Starting the module RNN C&C Detection (Detect C&C channels based on behavioral letters) [PID 83]
Starting the module Threat Intelligence (Check if the source IP or destination IP are in a malicious list of IPs) [PID 84]
Starting the module Timeline (Creates kalipso timeline of what happened in the network based on flows and available data) [PID 98]
Starting the module Update Manager (Update Threat Intelligence files) [PID 112]
Starting the module Virustotal (IP, domain and file hash lookup on Virustotal) [PID 113]
---------------------------
[Main] Disabled Modules: ['template', 'ensembling', 'exporting_alerts', 'p2ptrust', 'cesnet', 'blocking', 'leak_detector', 'cyst']
[Evidence] Storing Slips logs in output/test7-malicious.pcap_2024-07-17_09:16:11/
Started Evidence Process [PID 114]
Started Profiler Process [PID 115]
[Main] Metadata added to output/test7-malicious.pcap_2024-07-17_09:16:11/metadata
Started Input Process [PID 116]
[Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
[Input] Storing zeek log files in output/test7-malicious.pcap_2024-07-17_09:16:11/zeek_files
[Input] We read everything. No more input. Stopping input process. Sent 541 linesast 1 hr: 4. (2024/07/17 09:16:54)
[Input] Stopping. Total lines read: 541
[Main] nalyzed IPs so far: 101. Evidence Added: 205. IPs sending traffic in the last 1 hr: 0. (2024/07/17 09:16:59)
---------------------------
[Main] Stopping Slips
[Main] Analysis of dataset/test7-malicious.pcap finished in 0.76 minutes
[Main] Threat Intelligence Stopped. 13 left.
[Main] Profiler Stopped. 12 left.
[Main] RNN C&C Detection Stopped. 11 left.
[Main] HTTP Analyzer Stopped. 10 left.
[Main] Input Stopped. 9 left.
[Main] Timeline Stopped. 8 left.
[Main] ARP Stopped. 7 left.
[Main] Flow ML Detection Stopped. 6 left.
[Main] Progress Bar Stopped. 5 left.
[Main] Network Discovery Stopped. 4 left.
[Main] The following modules are busy working on your data.
['IP Info', 'Update Manager', 'Flow Alerts', 'Evidence']
You can wait for them to finish, or you can press CTRL-C again to force-kill.
[Main] Update Manager may take several minutes to finish updating 45+ TI files.
[Main] IP Info Stopped. 3 left.
[Main] Flow Alerts Stopped. 2 left.
[Update Manager] 43 TI files successfully loaded.
[Main] Update Manager Stopped. 1 left.
[Main] Evidence Stopped. 0 left.
There are 203 evidences in the output/test7-malicious.pcap_2024-07-17_09:16:11/alerts.log
I use ml train module and modify slips.conf to train mode. I tried, still has this problem.
Ah! thanks I was going to ask if you were in training mode. That is the issue to trigger. I confirm that I could reproduce the problem. Planning the fix
Hi! When I use mlflow train module and type
but I got response like
Error in process_features()
herehow should I do?