stratosphereips / StratosphereLinuxIPS

Slips, a free software behavioral Python intrusion prevention system (IDS/IPS) that uses machine learning to detect malicious behaviors in the network traffic. Stratosphere Laboratory, AIC, FEL, CVUT in Prague.
Other
686 stars 165 forks source link

ML train with error in process_features() #819

Open whale-withme opened 1 month ago

whale-withme commented 1 month ago

Hi! When I use mlflow train module and type

./slips -f dataset/test7-malicious.pcap -c config/slips.conf 

but I got response like Error in process_features() here

[Input] Storing zeek log files in output/test7-malicious.pcap_2024-07-16_02:55:19/zeek_files
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 51.0.
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 101.0.16 02:55:28)
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 151.0.
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 201.0.
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 251.0.
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 301.0.
[Flow ML Detection] Error in process_features()
[Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 351.0.07/16 02:55:33)
[Flow ML Detection] Error in process_features()
^C[Input] Stopping. Total lines read: 541

how should I do?

whale-withme commented 1 month ago

Not in pcap file, I tried interface and faced the same problem. Should I ignore this?

eldraco commented 1 month ago

Hi @whale-withme, please always report the versions of slips, commit and system you are using. If not it is impossible to know. You can also try -e 3 to have more debugging messages.

whale-withme commented 1 month ago

Hi! @eldraco I use slips version 1.0.15 and in master branch. Did it meet some data format conversion problem? I guess

whale-withme commented 1 month ago

Oh, my system is in docker. Sorry, I'm new.

eldraco commented 1 month ago

Can you provide the error.log file that slips generates? and maybe the whole slips.log too? it is usually in the output folder. In your case it would be something like output/test7-malicious.pcap/slips.log and similar for error.log

whale-withme commented 1 month ago

Sure! Here is error.log

2024/07/17 09:26:32.985524 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:32.985672 [Flow ML Detection] Error in train()
2024/07/17 09:26:32.985774 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:34.204108 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:34.204286 [Flow ML Detection] Error in train()
2024/07/17 09:26:34.204408 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:34.960821 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:34.960999 [Flow ML Detection] Error in train()
2024/07/17 09:26:34.961105 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:35.729694 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:35.729911 [Flow ML Detection] Error in train()
2024/07/17 09:26:35.730036 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:36.536441 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:36.536671 [Flow ML Detection] Error in train()
2024/07/17 09:26:36.536795 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:37.371597 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:37.371922 [Flow ML Detection] Error in train()
2024/07/17 09:26:37.372059 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:38.273439 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 140, in process_features
    dataset = dataset.drop(field, axis=1)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/frame.py", line 4906, in drop
    return super().drop(
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4150, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 4185, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/indexes/base.py", line 6017, in drop
    raise KeyError(f"{labels[mask]} not found in axis")
KeyError: "['flow_type'] not found in axis"

2024/07/17 09:26:38.273839 [Flow ML Detection] Error in train()
2024/07/17 09:26:38.273994 [Flow ML Detection] Traceback (most recent call last):
  File "/StratosphereLinuxIPS/modules/flowmldetection/flowmldetection.py", line 69, in train
    self.flows.label = self.flows.label.str.replace(
AttributeError: 'NoneType' object has no attribute 'label'

2024/07/17 09:26:52.434125 [Update Manager] Connection error while downloading the file https://check.torproject.org/torbulkexitlist. Aborting.

Here is slips.log

2024/07/17 09:26:25.390494 [Main] Using redis server on port: 6379
2024/07/17 09:26:25.390760 [Main] Started Main process [PID 3511]
2024/07/17 09:26:25.395186 [Main] Started PBar process [PID 3521]
2024/07/17 09:26:25.474654 [Main] Starting modules
2024/07/17 09:26:27.235406 [Main]       Starting the module ARP (Detect ARP attacks) [PID 3619]
2024/07/17 09:26:27.255412 [Main]       Starting the module Flow Alerts (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID 3622]
2024/07/17 09:26:27.265072 [Main]       Starting the module Flow ML Detection (Train or test a Machine Learning model to detect malicious flows) [PID 3623]
2024/07/17 09:26:27.275282 [Main]       Starting the module HTTP Analyzer (Analyze HTTP flows) [PID 3624]
2024/07/17 09:26:27.284780 [Main]       Starting the module IP Info (Get different info about an IP/MAC address) [PID 3625]
2024/07/17 09:26:27.293551 [Main]       Starting the module Network Discovery (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID 3626]
2024/07/17 09:26:27.303480 [Main]       Starting the module Risk IQ (Module to get passive DNS info about IPs from RiskIQ) [PID 3627]
2024/07/17 09:26:27.312780 [Main]       Starting the module RNN C&C Detection (Detect C&C channels based on behavioral letters) [PID 3628]
2024/07/17 09:26:27.345828 [Main]       Starting the module Threat Intelligence (Check if the source IP or destination IP are in a malicious list of IPs) [PID 3726]
2024/07/17 09:26:27.356497 [Main]       Starting the module Timeline (Creates kalipso timeline of what happened in the network based on flows and available data) [PID 3728]
2024/07/17 09:26:27.370486 [Main]       Starting the module Update Manager (Update Threat Intelligence files) [PID 3729]
2024/07/17 09:26:27.378817 [Update Manager] Checking if we need to download TI files.
2024/07/17 09:26:27.381869 [Main]       Starting the module Virustotal (IP, domain and file hash lookup on Virustotal) [PID 3730]
2024/07/17 09:26:27.882654 [Main] Disabled Modules: ['template', 'ensembling', 'exporting_alerts', 'p2ptrust', 'cesnet', 'blocking', 'leak_detector', 'cyst']
2024/07/17 09:26:27.890380 [Evidence] Storing Slips logs in output/test7-malicious.pcap_2024-07-17_09:26:23/
2024/07/17 09:26:27.897297 [Main] Started Evidence Process [PID 3731]
2024/07/17 09:26:27.909913 [Main] Started Profiler Process [PID 3732]
2024/07/17 09:26:27.912159 [Main] Metadata added to output/test7-malicious.pcap_2024-07-17_09:26:23/metadata
2024/07/17 09:26:27.921421 [Main] Started Input Process [PID 3733]
2024/07/17 09:26:27.924313 [Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
2024/07/17 09:26:27.931409 [Input] Storing zeek log files in output/test7-malicious.pcap_2024-07-17_09:26:23/zeek_files
2024/07/17 09:26:32.962922 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 51.0.
2024/07/17 09:26:32.981987 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:34.189028 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 101.0.
2024/07/17 09:26:34.203826 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:34.943573 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 151.0.
2024/07/17 09:26:34.960564 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:35.712896 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 201.0.
2024/07/17 09:26:35.729430 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:36.517875 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 251.0.
2024/07/17 09:26:36.536187 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:37.351208 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 301.0.
2024/07/17 09:26:37.371325 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:38.208675 [Flow ML Detection] Training the model with the last group of flows and labels. Total flows: 351.0.
2024/07/17 09:26:38.273124 [Flow ML Detection] Error in process_features()
2024/07/17 09:26:52.436421 [Update Manager] 43 TI files successfully loaded.
2024/07/17 09:27:00.941889 [Input] We read everything. No more input. Stopping input process. Sent 541 lines
2024/07/17 09:27:00.941984 [Input] Telling Profiler to stop because no more input is arriving.
2024/07/17 09:27:00.942044 [Input] Waiting for Profiler to stop.
2024/07/17 09:27:00.942222 [Profiler] Stopping. Total lines read: 541
2024/07/17 09:27:00.942342 [Profiler] Marking Profiler as done processing.
2024/07/17 09:27:00.942378 [Profiler] Profiler is done processing.
2024/07/17 09:27:00.942466 [Profiler] Profiler is done telling input.py that it's done processing.
2024/07/17 09:27:00.942499 [Profiler] Stopping. Total lines read: 541
2024/07/17 09:27:00.942485 [Input] Input is done processing.
2024/07/17 09:27:01.378260 [Input] Stopping. Total lines read: 541
2024/07/17 09:27:03.016316 [Main] 
---------------------------
2024/07/17 09:27:03.016557 [Main] Stopping Slips
2024/07/17 09:27:03.034812 [Main] Analysis of dataset/test7-malicious.pcap finished in 0.63 minutes
2024/07/17 09:27:03.035100 [Main] Total flows read (without altflows): 374
2024/07/17 09:27:03.035815 [Main]   Update Manager         Stopped. 13 left.
2024/07/17 09:27:03.035957 [Main]   Profiler               Stopped. 12 left.
2024/07/17 09:27:03.049347 [Main]   Timeline               Stopped. 11 left.
2024/07/17 09:27:03.342716 [Main]   IP Info                Stopped. 10 left.
2024/07/17 09:27:06.345985 [Main]   RNN C&C Detection      Stopped. 9 left.
2024/07/17 09:27:06.346397 [Main]   Threat Intelligence    Stopped. 8 left.
2024/07/17 09:27:06.346725 [Main]   ARP                    Stopped. 7 left.
2024/07/17 09:27:06.347016 [Main]   Flow ML Detection      Stopped. 6 left.
2024/07/17 09:27:06.347293 [Main]   HTTP Analyzer          Stopped. 5 left.
2024/07/17 09:27:06.347548 [Main]   Network Discovery      Stopped. 4 left.
2024/07/17 09:27:06.347819 [Main]   Input                  Stopped. 3 left.
2024/07/17 09:27:06.348077 [Main]   Progress Bar           Stopped. 2 left.
2024/07/17 09:27:06.348283 [Main] The following modules are busy working on your data.

['Flow Alerts', 'Evidence']

You can wait for them to finish, or you can press CTRL-C again to force-kill.

2024/07/17 09:27:19.942328 [Main]   Evidence               Stopped. 1 left.
2024/07/17 09:27:19.964624 [Main]   Flow Alerts            Stopped. 0 left.
2024/07/17 09:27:19.968192 [Main] [Process Manager] Slips didn't shutdown gracefully - User pressed ctr+c or Slips was killed by the OS
eldraco commented 1 month ago

In Linux with Docker, for us it works without errors.

Linux 3c0d724d10ba 5.15.0-75-generic #82~20.04.1-Ubuntu Docker version 27.0.3, build 7d4bcd8

./slips.py -f dataset/test7-malicious.pcap -c config/slips.conf -e 1

https://stratosphereips.org
---------------------------
[Main] Using redis server on port: 6379
Started Main process [PID 31]
Started PBar process [PID 48]
Starting modules
        Starting the module ARP (Detect ARP attacks) [PID 74]
        Starting the module Flow Alerts (Alerts about flows: long connection, successful ssh, password guessing, self-signed certificate, data exfiltration, etc.) [PID 77]
        Starting the module Flow ML Detection (Train or test a Machine Learning model to detect malicious flows) [PID 78]
        Starting the module HTTP Analyzer (Analyze HTTP flows) [PID 79]
        Starting the module IP Info (Get different info about an IP/MAC address) [PID 80]
        Starting the module Network Discovery (Detect Horizonal, Vertical Port scans, ICMP, and DHCP scans) [PID 81]
        Starting the module Risk IQ (Module to get passive DNS info about IPs from RiskIQ) [PID 82]
        Starting the module RNN C&C Detection (Detect C&C channels based on behavioral letters) [PID 83]
        Starting the module Threat Intelligence (Check if the source IP or destination IP are in a malicious list of IPs) [PID 84]
        Starting the module Timeline (Creates kalipso timeline of what happened in the network based on flows and available data) [PID 98]
        Starting the module Update Manager (Update Threat Intelligence files) [PID 112]
        Starting the module Virustotal (IP, domain and file hash lookup on Virustotal) [PID 113]
---------------------------
[Main] Disabled Modules: ['template', 'ensembling', 'exporting_alerts', 'p2ptrust', 'cesnet', 'blocking', 'leak_detector', 'cyst']
[Evidence] Storing Slips logs in output/test7-malicious.pcap_2024-07-17_09:16:11/
Started Evidence Process [PID 114]
Started Profiler Process [PID 115]
[Main] Metadata added to output/test7-malicious.pcap_2024-07-17_09:16:11/metadata
Started Input Process [PID 116]
[Main] Warning: Slips may generate a large amount of traffic by querying TI sites.
[Input] Storing zeek log files in output/test7-malicious.pcap_2024-07-17_09:16:11/zeek_files
[Input] We read everything. No more input. Stopping input process. Sent 541 linesast 1 hr: 4. (2024/07/17 09:16:54)
[Input] Stopping. Total lines read: 541
[Main] nalyzed IPs so far: 101. Evidence Added: 205. IPs sending traffic in the last 1 hr: 0. (2024/07/17 09:16:59)
---------------------------
[Main] Stopping Slips
[Main] Analysis of dataset/test7-malicious.pcap finished in 0.76 minutes
[Main]  Threat Intelligence     Stopped. 13 left.
[Main]  Profiler                Stopped. 12 left.
[Main]  RNN C&C Detection       Stopped. 11 left.
[Main]  HTTP Analyzer           Stopped. 10 left.
[Main]  Input                   Stopped. 9 left.
[Main]  Timeline                Stopped. 8 left.
[Main]  ARP                     Stopped. 7 left.
[Main]  Flow ML Detection       Stopped. 6 left.
[Main]  Progress Bar            Stopped. 5 left.
[Main]  Network Discovery       Stopped. 4 left.
[Main] The following modules are busy working on your data.

['IP Info', 'Update Manager', 'Flow Alerts', 'Evidence']

You can wait for them to finish, or you can press CTRL-C again to force-kill.

[Main] Update Manager may take several minutes to finish updating 45+ TI files.
[Main]  IP Info                 Stopped. 3 left.
[Main]  Flow Alerts             Stopped. 2 left.
[Update Manager] 43 TI files successfully loaded.
[Main]  Update Manager          Stopped. 1 left.
[Main]  Evidence                Stopped. 0 left.

There are 203 evidences in the output/test7-malicious.pcap_2024-07-17_09:16:11/alerts.log

whale-withme commented 1 month ago

I use ml train module and modify slips.conf to train mode. I tried, still has this problem.

eldraco commented 1 month ago

Ah! thanks I was going to ask if you were in training mode. That is the issue to trigger. I confirm that I could reproduce the problem. Planning the fix