search

koumajos / Classification_by_NetTiSA_flow

Classification of Network Traffic based NetTiSA flow
4 stars 1 forks source link

Python code for converting data #2

Open tonglingxiangyu opened 2 days ago

tonglingxiangyu commented 2 days ago

Hello, I hope you can provide the python code for converting the original csv file of the dataset to the csv file of the 20 features selected in the article. This will be of great help to my research!

koumajos commented 2 days ago

Hello, I unfortunately cannot. Because the dataset CSV files are created using IP flow exporter, which processed the original dataset PCAP files. The dataset CSV files cannot be created from the original dataset CSV files.

By now, the NetTiSA feature set has been implemented in the Ipfixprobe IP flow exporter, which can be easily used to process PCAP files to obtain IP flows with our features. Moreover, you can mine the C++ code, which processes each packet and computes the NetTiSA features. The code is here: https://github.com/CESNET/ipfixprobe/blob/master/process/nettisa.cpp

Function void NETTISAPlugin::update_record(RecordExtNETTISA* nettisa_data, const Packet& pkg, const Flow& rec) takes a new packet and updates the features. The function void NETTISAPlugin::pre_export(Flow& rec) is processed before exporting the IP flow (flows are exported when inactive or active timeout is placed). Thus, you can load packets into, for example, Python, then split the packets on connections, then on packet sequences by active and inactive timeouts, and finally compute the NetTiSA features.

tonglingxiangyu commented 2 days ago

Hello, I unfortunately cannot. Because the dataset CSV files are created using IP flow exporter, which processed the original dataset PCAP files. The dataset CSV files cannot be created from the original dataset CSV files.

By now, the NetTiSA feature set has been implemented in the Ipfixprobe IP flow exporter, which can be easily used to process PCAP files to obtain IP flows with our features. Moreover, you can mine the C++ code, which processes each packet and computes the NetTiSA features. The code is here: https://github.com/CESNET/ipfixprobe/blob/master/process/nettisa.cpp

Function void NETTISAPlugin::update_record(RecordExtNETTISA* nettisa_data, const Packet& pkg, const Flow& rec) takes a new packet and updates the features. The function void NETTISAPlugin::pre_export(Flow& rec) is processed before exporting the IP flow (flows are exported when inactive or active timeout is placed). Thus, you can load packets into, for example, Python, then split the packets on connections, then on packet sequences by active and inactive timeouts, and finally compute the NetTiSA features.

Thank you very much. May I ask what are the steps to use Ipfixprobe and what are the specific configuration and parameters? I hope I can reproduce this process locally.

koumajos commented 2 days ago

I recommend that you use our docker https://github.com/CESNET/ipfixprobe/tree/master/docker

Following the README, you can process the PCAP file and obtain the CSV file with one command in the terminal. The CSV will contain NetTiSA features (the third set of NetTiSA features you must compute in postprocessing on your own, for example, in Python) and the PSTATS features (packet sequences for first 30 packets of IP flow -- in literature sometimes referred to as SPLT). If you want some different feature sets (from these: https://cesnet.github.io/ipfixprobe//export/), you can add a feature set using -p argument in this file: https://github.com/CESNET/ipfixprobe/blob/master/docker/process_script.sh

tonglingxiangyu commented 2 days ago

I recommend that you use our docker https://github.com/CESNET/ipfixprobe/tree/master/docker

Following the README, you can process the PCAP file and obtain the CSV file with one command in the terminal. The CSV will contain NetTiSA features (the third set of NetTiSA features you must compute in postprocessing on your own, for example, in Python) and the PSTATS features (packet sequences for first 30 packets of IP flow -- in literature sometimes referred to as SPLT). If you want some different feature sets (from these: https://cesnet.github.io/ipfixprobe//export/), you can add a feature set using -p argument in this file: https://github.com/CESNET/ipfixprobe/blob/master/docker/process_script.sh

I will try to learn how to use Ipfixprobe, thanks for your answer.

tonglingxiangyu commented 2 days ago

I recommend that you use our docker https://github.com/CESNET/ipfixprobe/tree/master/docker

Following the README, you can process the PCAP file and obtain the CSV file with one command in the terminal. The CSV will contain NetTiSA features (the third set of NetTiSA features you must compute in postprocessing on your own, for example, in Python) and the PSTATS features (packet sequences for first 30 packets of IP flow -- in literature sometimes referred to as SPLT). If you want some different feature sets (from these: https://cesnet.github.io/ipfixprobe//export/), you can add a feature set using -p argument in this file: https://github.com/CESNET/ipfixprobe/blob/master/docker/process_script.sh

Which datasets can directly calculate the 20 features from the csv file without having to calculate from the pcap file?

koumajos commented 2 days ago

I think none of them. I compute all the datasets from the PCAP files.

tonglingxiangyu commented 2 days ago

I think none of them. I compute all the datasets from the PCAP files.

I think none of them. I compute all the datasets from the PCAP files.

OK, thanks. When using Ipfixprobe to extract pcap packet features, how should the configuration file be configured and how should the parameters be set so that the csv file obtained has the features we need? Sorry, I am not very familiar with this tool, and I hope there is an operation manual to learn from.