search

ahlashkari / CICFlowMeter

CICFlowmeter-V4.0 (formerly known as ISCXFlowMeter) is an Ethernet traffic Bi-flow generator and analyzer for anomaly detection that has been used in many Cybersecurity datsets such as Android Adware-General Malware dataset (CICAAGM2017), IPS/IDS dataset (CICIDS2017), Android Malware dataset (CICAndMal2017) and Distributed Denial of Service (CICDDoS2019).
Other
537 stars 269 forks source link

Possible miscalculations of flow bulk, sub flow and TCP flags-related features. #41

Closed lnutimura closed 5 years ago

lnutimura commented 5 years ago

Greetings,

I noticed that a week ago you guys changed the "BasicFlow.java" file with the commit ed1d6bcbdefe161be51ac278f76be995f7121482. In this specific commit, three functions were uncommented:

updateFlowBulk(packet);
detectUpdateSubflows(packet);
checkFlags(packet);

Which in this case, I believe, means that they weren't taken into account before, when updating existent flows with the addPacket(BasicPacketInfo packet) method, only with the firstPacket(BasicPacketInfo packet).

Since I'm using the CICIDS2017's .csv files to train a few machine learning techniques and these files were previously generated by the CICFlowMeter, I'm concerned about possible miscalculations of certain features related to these three methods in the files provided w/ the dataset for machine learning purposes.

sword1996 commented 5 years ago

I also have the same doubts about this code.

Further, I think there is a problem with the TCP Flag Count traffic characteristics and the DDoS label in the CICIDS2017's .csv data set (why the TCP Flag Count value with DDoS tag data is only 0 or 1, does not meet the characteristics of DDoS) .

Another strange problem is that the code also comments the following code,

// Flow finished due FIN flag (tcp only):

            }else if(packet.hasFlagFIN()){
                ...
                if (mListener != null) {

                } /*else {
                    finishedFlows.put(getFlowCount(), flow);
                }*/

                ....
            }

meaning that the code will only deal with the characteristics of the traffic data left in memory, Will no longer process the data in 'finishedFlows'.

Do you have the same problem?

Thanks to the open source code given by this lab. : )

ahlashkari commented 5 years ago

Hi Guys, on the previous version, it just checks the flags (0 or 1) but in the new version, it is counting the flags. If the number of flags is an important feature for your learning system, pls recalculate them.

ahlashkari commented 6 months ago

Hi, We are pleased to announce that the first version of the Network and Transportation Layers Flow Analyzer (NTLFlowLyzer) is now available as part of the Understanding Cybersecurity Series (UCS) knowledge mobilization program. This Python open-source project has been designed and developed to address many issues encountered with CICFlowMeter. It extracts over 300 features from TCP-based network traffic, tailored explicitly for Anomaly Profiling (AP). NTLFlowLyzer serves as a key component of the upcoming NetFlowLyzer.

https://github.com/ahlashkari/NTLFlowLyzer

Best, BCCC Team https://www.yorku.ca/research/bccc/