search

ahlashkari / CICFlowMeter

CICFlowmeter-V4.0 (formerly known as ISCXFlowMeter) is an Ethernet traffic Bi-flow generator and analyzer for anomaly detection that has been used in many Cybersecurity datsets such as Android Adware-General Malware dataset (CICAAGM2017), IPS/IDS dataset (CICIDS2017), Android Malware dataset (CICAndMal2017) and Distributed Denial of Service (CICDDoS2019).
Other
519 stars 266 forks source link

Unable to obtain UDP Flows #13

Closed ameya-joshi-tuc closed 5 years ago

ameya-joshi-tuc commented 5 years ago

Hello

I am unable to obtain UDP Flows using CICFlowMeter in both Online/Offline mode. I made a pcap capture with Wireshark to try and extract UDP flows using CICFlowMeter but it only creates the TCP flows and not the UDP ones. Also, is there a way to obtain ICMP flows between two IP addresses. I mostly use Python, not Java so I don't know how or where I can make changes for controlling UDP timeouts and capturing ICMP flows but if someone can point me in the right direction, I will be extremely grateful. More important is the UDP issue. (I unfortunately cannot upload the exact files)

All captures are carried out on a Virtual Machine running Ubuntu 16.04 LTS

I thank you for your help

kshntn commented 5 years ago

Hi @AmJo29 I am having exactly the same issue. I am trying to generate UDP and also want to generate features for packets which are not complete. But I am getting zero packets generated for those! Currently, I see it working only for packets which are complete!

imanoracle commented 5 years ago

@kshntn @AmJo29 Please upload your sample PCAP in order to investigate more (I do not have any problem with UDP). @kshntn The logic for closing a Netflow is defined in addPacket function in FlowGenerator.java. In order to capture flows that do not have FIN flag (packet.hasFlagFIN()), you should reduce the Netflow timeout (currently, it is 2 minutes). You can find it on line 67 of CICFlowMeter.java file: flowGen = new FlowGenerator(true,120000000L, 5000000L);

kshntn commented 5 years ago

Hi @imanoracle I generated a pcap file for a DDOS attack.. It has packets which is not complete. I have uploaded it here. Also I will send the .pcap for udp in some time! notworking.pcap.zip Can you tell me if its working. Or is it the issue of the pcap file!

ameya-joshi-tuc commented 5 years ago

@imanoracle Thank you for the reply. I cannot upload the files unfortunately. I did experiment with the UDP Timeouts in offline mode and it did not work, but thanks for pointing out the line number in the file, I will see if that helps me with the online phase. Maybe if it turns out that the problem is in the PCAP files (@kshntn ), I will generate a new set of packets on a different system and see if that helps

kshntn commented 5 years ago

Hi @AmJo29 I had some UDP packets in a pcap file. but the features were only generated for the bidirectional TCP packets!

ameya-joshi-tuc commented 5 years ago

Yes that is the issue for me as well. Since UDP doesn't include the usual TCP handshakes for establishing a connection and FIN (As indicated by @imanoracle) to terminate the connection, we can only give it timeouts and ask it to calculate the flow statistics for 'that' timeout window between a source and a destination. Hasn't worked for me though sadly

kshntn commented 5 years ago

Hi @AmJo29 thats right. I tried the way @imanoracle mentioned, but it was not successful.. I guess my pcap file has some errors or I did not do it right! Could you please tell me whether the pcap file i uploaded is okay ? Can I have some way to contact you faster (email id or so?).

ameya-joshi-tuc commented 5 years ago

I think GitHub is probably the fastest way tbh. Also, there are no UDP packets in your PCAP file

kshntn commented 5 years ago

No @AmJo29 .. Not this one.. This one is for DDOS attacks using TCP SYN flood. This one has packets which are not complete. I wanted features for this too!

working.pcap.zip This one is a pcap with TCP and UDP packets and this generates 30 flows which are only for bidirectional TCP packets. Could you check it at your end too!

ameya-joshi-tuc commented 5 years ago

@kshntn I was able to generate the UDP flows for your PCAP file (working.pcap) I took a random UDP timestamp (at around time 20) The next non-UDP flow occurs at timestamp 25 so in CICFlowMeter, I set the Timeout values to 6000000 (Both timeout fields; the timeout unit is in microseconds) and was able to generate 82 Flows which includes both UDP and TCP. I confirmed it was the same flow by checking the source and destination addresses and ports but did not have the time to verify if the values generated by CICFlowMeter are correct (you can try that out). Try the steps I mentioned at your end and see if it works for you, if not maybe you will have to reinstall the application or something

kshntn commented 5 years ago

Hi @AmJo29 I tried changing the timeout values to 6000000 and then tried creating CSV files. The number of flows was changing each time I executed it. Is that expected?

ameya-joshi-tuc commented 5 years ago

If it is maintained at 6000000, according to me it should not. If you are changing the values, then yes it can change. I suggest you go over the CICFlowMeter and related papers, you will understand it better - http://www.netflowmeter.ca/netflowmeter.html

kshntn commented 5 years ago

Thanks @AmJo29 I will go through it and let you know.. Did you change the timeout values in the code or the directly in the tool?

ameya-joshi-tuc commented 5 years ago

Since I checked your PCAP files which is offline mode, I changed it with the tool. If you need it real-time, you will have to modify the code Try and see if the changes work for you. It hasn't worked for me still but if it works for you, the issue might be at my end, so do post if it worked and what changes you did so that the issue can be marked with your solution as closed by the mods (I am new to GitHub, I do not know how this works)

kshntn commented 5 years ago

I changed it in the code. I will try again and let you know. But I am also looking for features for pcap with incomplete packets. Can you also tell me if the notworking pcap which I uploaded some time back can also be used to generate features?

ameya-joshi-tuc commented 5 years ago

See the links, you will understand. The best way is to simply experiment and see what all works. Since I am a fellow user like you and not the developer, there is only so much that I know and can help you with :)

kshntn commented 5 years ago

Thank you @AmJo29 . I am going through the papers.. :+1:

ameya-joshi-tuc commented 5 years ago

Mention not. All the best and do post a solution if the problem is solved so that the mods can close the thread :)

MZarea commented 5 years ago

Hi!

I have one question related to extracting .CSV dataset from PCAP file, in order to be able to continue my work. I will greatly appreciate your answer.

I need to extract time series dataset from pcap file with a time window of 10 minutes.

would you help me how change the time windows in CICFlowMeter source code? and whats the default value?

Also, are you have any idea if there any option in TSHARK or tranalyzer to do the same?

Mahmoud

ahlashkari commented 4 months ago

Hi, We are pleased to announce that the first version of the Network and Transportation Layers Flow Analyzer (NTLFlowLyzer) is now available as part of the Understanding Cybersecurity Series (UCS) knowledge mobilization program. This Python open-source project has been designed and developed to address many issues encountered with CICFlowMeter. It extracts over 300 features from TCP-based network traffic, tailored explicitly for Anomaly Profiling (AP). NTLFlowLyzer serves as a key component of the upcoming NetFlowLyzer.

https://github.com/ahlashkari/NTLFlowLyzer

Best, BCCC Team https://www.yorku.ca/research/bccc/