fklassen
commented
9 years ago Received the following ...
Dear Fred and Aaron, I can say that the Memory Leaking issue is not related to tcpreplay because when I opened the 11 GB pcap file by Wireshark, the memory also reach till 32 GB. I think the issue is either with the file or with hardware.
As per email thread:
Hi, I'm Aid, My tesbed: I run tcpreply on a PC1 (ubuntu 12.04 with Dell, i7 CPU, and 32 RAM, 1Gbps NIC) on eth0 which is connected to a Giga switch which forward the traffic (using port mirror) to another PC2 for traffic analysis. Pcap size: 11 GB
First, I run the following command tcpreplay -i eth0 -M 100 data.pcap Then after few seconds suddenly eth0 become down. I need to use ifdown and ifup then it works fine, then again down and so on. 1) What is the cause? and How to solve this issue?
Then I run the following command (just add -K) tcpreplay -i eth0 -K -M 100 data.pcap It works and eth0 is not hanging, but RAM (with 32 GB although the pcap size is 11 GB) suddenly filled from 5 MB to 32 GB full and system hang (monitored by htop). 2) What is the cause? and How to solve this issue?
I want to replay the pcap once only
I hope someone answer the two questions above, thanks
Aaron Turner via lists.sourceforge.net Jun 17 (5 days ago)
to Main First problem is probably a network driver or other hardware issue (possibly the switch itself).
Second problem sounds like a memory leak. Maybe. Dunno, since you didn't say what version of tcpreplay you're using nobody can help you.
Side note: mirror/SPAN ports often drop packets.
Aaron Turner http://synfin.net/ Twitter: @synfinatic Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin
Tcpreplay-users mailing list Tcpreplay-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tcpreplay-users Support Information: http://tcpreplay.synfin.net/trac/wiki/Support
Hashem Alaidaros
to Main Here is additional information: Tcpreplay: tcpreplay version: 4.1.0 (build git:v4.1.0) Cache file supported: 04 Not compiled with libdnet. Compiled against libpcap: 1.1.1 64 bit packet counters: enabled Packet editing: disabled Fragroute engine: disabled Injection method: PF_PACKET send() Not compiled with netmap
Pcap: PCAP file: http://www.isot.ece.uvic.ca/dataset/ISOT_Botnet_DataSet_2010.tar.gz
NIC: Ethernet controller: Intel Corporation Device 153a (rev 04) Subsystem: Dell Device 05a4 Kernel driver in use: e1000e Kernel modules: e1000e
I hope I get the answers of the two questions sent in the previous post.
A friend in need Is a friend indeed
Tcpreplay-users mailing list Tcpreplay-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tcpreplay-users Support Information: http://tcpreplay.synfin.net/trac/wiki/Support
Aaron Turner via lists.sourceforge.net Jun 17 (5 days ago)
to Main I'll let Fred answer the memory leak issue, but since you're using the PF_PACKET send() method for sending packets, I can't think of any way it's tcpreplay's fault your NIC is going down. You've got a problem at a lower level (kernel, driver, hardware, switch, etc). You'll have to debug that yourself. :(
Fredrick Klassen
to Main Yes, I agree with Aaron. It is difficult to imagine tcpreplay being the cause of the memory leak when using PF_PACKET.
How are you seeing the memory leak? Are you using “top”? If so, can you send me a screen capture that includes the offending process? Also, please have a look at either syslog or the kernel log for any trace info. I am specifically looking for OOM (out-of-memory) logs. Finally, please send me the output of the ‘lsmod’ and the ‘uname -a’ commands.
Thanks, Fred.
Aaron Turner Jun 18 (4 days ago)
to me I think you mean "network going down" not memory leak. Memory leak is probably due to a bug with caching the packets in memory.
Hashem Alaidaros aidaros.dev@gmail.com via lists.sourceforge.net AttachmentsJun 19 (3 days ago)
to Main Thanks Aaron and Fredrick for reply, I was really waiting for solving my issue. (I sent this email yesterday, but it rejected my the moderator due to size limit exceed, then I reduce the file size)
I use "htop" to see the memory usage. Just for your information: This PC has two ethernets, one for tcpreplay use, the other is for teamviewer (remote acccess from Internet) use.
1) I attached you screen capture after run the command: sudo tcpreplay -i eth0 -K -M 100 dataset1.pcap, (before I run the command the memory usage was 500 MB, after run the command it jumps to 32 GB although the pcap file is 11 GB)
2) I attached the kern.log (as u can see there is no log written for today, there is log for only yesterday)
3) And Here is the last part of syslog:
Jun 18 20:54:50 hashem-OptiPlex-9020 dhclient: bound to 10.10.2.51 -- renewal in 1518 seconds. Jun 18 21:17:02 hashem-OptiPlex-9020 CRON[8191]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jun 18 21:20:08 hashem-OptiPlex-9020 dhclient: DHCPREQUEST of 10.10.2.51 on eth1 to 10.10.2.21 port 67 Jun 18 21:20:08 hashem-OptiPlex-9020 dhclient: DHCPACK of 10.10.2.51 from 10.10.2.21 Jun 18 21:20:08 hashem-OptiPlex-9020 dhclient: bound to 10.10.2.51 -- renewal in 1558 seconds. Jun 18 21:46:06 hashem-OptiPlex-9020 dhclient: DHCPREQUEST of 10.10.2.51 on eth1 to 10.10.2.21 port 67 Jun 18 21:46:06 hashem-OptiPlex-9020 dhclient: DHCPACK of 10.10.2.51 from 10.10.2.21 Jun 18 21:46:06 hashem-OptiPlex-9020 dhclient: bound to 10.10.2.51 -- renewal in 1587 seconds. Jun 18 21:50:49 hashem-OptiPlex-9020 pulseaudio[2417]: [pulseaudio] module-console-kit.c: GetUnixUser() call failed: org.freedesktop.DBus.Error.UnknownMethod: Method "GetUnixUser" with signature "" on interface "org.freedesktop.ConsoleKit.Session" doesn't exist Jun 18 22:12:33 hashem-OptiPlex-9020 dhclient: DHCPREQUEST of 10.10.2.51 on eth1 to 10.10.2.21 port 67 Jun 18 22:12:33 hashem-OptiPlex-9020 dhclient: DHCPACK of 10.10.2.51 from 10.10.2.21 Jun 18 22:12:33 hashem-OptiPlex-9020 dhclient: bound to 10.10.2.51 -- renewal in 1707 seconds.
Jun 18 22:17:01 hashem-OptiPlex-9020 CRON[8394]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
4) Output of uname -a Linux hashem-OptiPlex-9020 3.5.0-44-generic #67~precise1-Ubuntu SMP Wed Nov 13 16:16:57 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
5) I attached the lsmod output
Thanks
Tcpreplay-users mailing list Tcpreplay-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tcpreplay-users Support Information: http://tcpreplay.synfin.net/trac/wiki/Support 3 Attachments
Preview attachment After run tcpreplay2.jpg Image After run tcpreplay2.jpg Preview attachment Kern-log.txt
Text Kern-log.txt Preview attachment lsmod.txt
Text lsmod.txt
Hashem Alaidaros to Main Hello Aaron and Fredrick, Just to inform you that, I attached the following in the previous email I sent on 19 June 2015,
1) screen capture after run the command: sudo tcpreplay -i eth0 -K -M 100 dataset1.pcap, (before I run the command the memory usage was 500 MB, after run the command it jumps to 32 GB although the pcap file is 11 GB) 2) kern.log 3) the last part of syslog 4) Output of uname -a 5) the lsmod output
Please inform when additional information is needed
Your anticipated help is approciated
Thanks
A friend in need Is a friend indeed
Tcpreplay-users mailing list Tcpreplay-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tcpreplay-users Support Information: http://tcpreplay.synfin.net/trac/wiki/Support
Aaron Turner via lists.sourceforge.net Jun 21 (1 day ago)
to Main I haven't had a chance to figure out the memory leak, but it occurs to me, I don't understand why you are caching the pcap in memory? You're only running the traffic at 100Mbps... Even a 7200 RPM disk should be
able to sustain 12.5 MB/sec sequential reads without any problem.
Aaron Turner http://synfin.net/ Twitter: @synfinatic Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin