Simple tcpliveplay test

[riramar]

I'm trying a simple test with tcpliveplay but it seems tcpliveplay doesn't handle TCP handshake properly. Let me describe the steps from my testing.

1. Logged as root on machine A I ran this command to capture all the traffic to the host 104.31.95.22 (my website).

# tcpdump host 104.31.95.22 -n -s 65535 -w http.pcap

2. In another shell instance under the same machine I performed a simple GET request with the command below.

# curl http://104.31.95.22

3. After curl finished I stopped tcpdump with Ctrl+C and noticed that tcpdump capture 9 packets.

# tcpdump host 104.31.95.22 -n -s 65535 -w http.pcap
tcpdump: listening on enp0s25, link-type EN10MB (Ethernet), capture size 65535 bytes
^C9 packets captured
9 packets received by filter
0 packets dropped by kernel

4. I opened http.pcap just to see if everything goes fine in my GET request and 302 (redirect) response.

5. Copied http.pcap to machine B in order to try the tcpliveplay. To get the MAC address of my gw in this new network I ran the command below.

root@zion:~/Temp# ip n
192.168.1.1 dev eth0 lladdr e0:41:36:f3:01:b8 REACHABLE

6. Started wireshark on machine B and ran the command below in a shell.

root@zion:~/Temp# tcpliveplay eth0 http.pcap 104.31.95.22 e0:41:36:f3:01:b8 random
new source port:: 56760
Random Local SEQ: 1743999685
Packets Scheduled 9
Sending Local Packet............... [1]
Receiving Packets from remote host...
Received Remote Packet...............   [2]
Remote Pakcet Expectation met.
Proceeding in replay....

======================================================================
= TIMEOUT:: Remote host is not responding. You may have crashed      =
= the host you replayed these packets against OR the packet sequence =
= changed since the capture was taken resulting in differing         =
= expectations. Closing replay...                                    =
======================================================================

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ Unfortunately an error has occurred  halting the replay of  
~ the pcap file 'http.pcap'. Please see error above for details...   
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

----------------TCP Live Play Summary----------------
- Packets Scheduled to be Sent & Received:          9   
- Actual Packets Sent & Received:                   2   
- Total Local Packet Re-Transmissions due to packet       
- loss and/or differing payload size than expected: 0   
- Thank you for Playing, Play again!                      
----------------------------------------------------------

7. As you can see from the output didn't work. Let's take a look what wireshark captured.

8. tcpliveplay started the TCP handshake sending the SYN packed and received the SYN+ACK back but it doesn't finish handshake with ACK. The server try one more time with a TCP retransmission package but no response from tcpliveplay. In order to be sure that I can perform a GET to 104.31.95.22 from machine B I performed the same command (curl http://104.31.95.22) from machine B and record the traffic on wireshark.

9. The handshake is properly executed as you can see from the screenshot above.

Could you please help me to perform this simple test?

Thanks! Ricardo Iramar

[fklassen]

Thanks for outlining the problem.

I'm labeling this as a bug for now, but will not assign to myself. Tcpliveplay was developed through a Cisco project before I was involved in Tcpreplay. I'll see if I can find the original author and see if he is interested in maintaining this code.

If not, I'll either have to dive into it myself, get another volunteer to step in or make some drastic decisions about the future of Tcpliveplay.

[riramar]

Thanks a lot for the quick replay. Do you another way or tool that I could accomplish the same goal as tcpliveplay?

[fklassen]

I would suggest trying tcpliveplay version 3.4.4. I think that was the last version touched only be original developer. Mainly security fixes have gone in since then. If it does work for you, let me know and I can do git bisect to isolate the failing commit.

[riramar]

Thanks! I'll check and let you know.

[riramar]

The Ubuntu tcpreplay package version 3.4.4-2+deb8u1build0.16.04.1 doesn't have the tcpliveplay file. Are you sure this is the right version?

# apt show tcpreplay
Package: tcpreplay
Version: 3.4.4-2+deb8u1build0.16.04.1
Priority: optional
Section: universe/net
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Noël Köthe <noel@debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 1,032 kB
Depends: libc6 (>= 2.14), libpcap0.8 (>= 0.9.8)
Homepage: http://tcpreplay.synfin.net/
Download-Size: 198 kB
APT-Manual-Installed: yes
APT-Sources: http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Description: Tool to replay saved tcpdump files at arbitrary speeds
 Tcpreplay is aimed at testing the performance of a NIDS by
 replaying real background network traffic in which to hide
 attacks. Tcpreplay allows you to control the speed at which the
 traffic is replayed, and can replay arbitrary tcpdump traces. Unlike
 programmatically-generated artificial traffic which doesn't
 exercise the application/protocol inspection that a NIDS performs,
 and doesn't reproduce the real-world anomalies that appear on
 production networks (asymmetric routes, traffic bursts/lulls,
 fragmentation, retransmissions, etc.), tcpreplay allows for exact
 replication of real traffic seen on real networks.

N: There is 1 additional record. Please use the '-a' switch to see it
# dpkg-query -L tcpreplay
/.
/usr
/usr/bin
/usr/bin/tcprewrite
/usr/bin/tcpreplay-edit
/usr/bin/tcpprep
/usr/bin/tcpreplay
/usr/bin/tcpbridge
/usr/share
/usr/share/doc
/usr/share/doc/tcpreplay
/usr/share/doc/tcpreplay/copyright
/usr/share/doc/tcpreplay/changelog.Debian.gz
/usr/share/doc/tcpreplay/TODO.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/tcprewrite.1.gz
/usr/share/man/man1/tcpreplay-edit.1.gz
/usr/share/man/man1/tcpprep.1.gz
/usr/share/man/man1/tcpbridge.1.gz
/usr/share/man/man1/tcpreplay.1.gz

[fklassen]

@riramar my mistake. I started maintaining (and now own) 4.0.0 and didn't realize that tcpliveplay was not distributed with 3.4.4. I am now the lone maintainer of Tcpreplay and don't know the entire history of Tcpliveplay. I'll ask the previous maintainers what version it was supported.

I suspect support was dropped before 3.4.4, but the source code was not in the release tarballs. I may have to drop it as well, unless I can fix this bug.

Sorry, I don't know of an alternative. I think I saw something once in GitHub, but I cannot remember where.

[riramar]

No problem @fklassen ... I really appreciate your quick replies.

[fogs]

I am having the exact same issue on Kali with this version:

tcpliveplay version: 4.3.2 (build git:v4.3.2) (debug)
Copyright 2012 by Yazan Siam <tcpliveplay@gmail.com>
Not compiled with libdnet.
Compiled against libpcap: 1.9.1
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Injection method: PF_PACKET send()

Is there any news to this topic? @riramar did you solve the situation for yourself?

Thanks a bunch!

[fogs]

I have just downloaded and built the latest version from github and fail also with the other available injection methods:

tcpliveplay version: 4.3.2 (build git:v4.3.2)
Copyright 2012 by Yazan Siam <tcpliveplay@gmail.com>
Not compiled with libdnet.
Compiled against libpcap: 1.9.1
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Injection method: pcap_sendpacket()

and

tcpliveplay version: 4.3.2 (build git:v4.3.2)
Copyright 2012 by Yazan Siam <tcpliveplay@gmail.com>
Not compiled with libdnet.
Compiled against libpcap: 1.9.1
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Injection method: pcap_inject()

Here is the output of one run:

root@kali:~/Downloads/tcpreplay-4.3.2# src/tcpliveplay eth0 ~/service-connection.pcap 192.168.0.5 re:dac:ted:6C:CB:BB random
new source port:: 59506
Random Local SEQ: 140580199
Packets Scheduled 241
Sending Local Packet...............     [1]
Receiving Packets from remote host...
Received Remote Packet...............   [2]
Remote Pakcet Expectation met.
Proceeding in replay....

======================================================================
= TIMEOUT:: Remote host is not responding. You may have crashed      =
= the host you replayed these packets against OR the packet sequence =
= changed since the capture was taken resulting in differing         =
= expectations. Closing replay...                                    =
======================================================================

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ Unfortunately an error has occurred  halting the replay of
~ the pcap file '/root/service-connection.pcap'. Please see error above for details...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

----------------TCP Live Play Summary----------------
- Packets Scheduled to be Sent & Received:          241
- Actual Packets Sent & Received:                   2
- Total Local Packet Re-Transmissions due to packet
- loss and/or differing payload size than expected: 0
- Thank you for Playing, Play again!
----------------------------------------------------------

What I can see in a parallel wireshark session is, that my machine is sending out a TCP SYN packet and the server is responding with a SYN ACK. No ACK is given by my machine. After a few seconds, the server resends the SYN ACK a couple of times. Finally the program aborts, claiming that the server did not respond. Opening a TCP session with telnet to the same port works just fine. There are no iptables rules, except for the one suppressing the RST from the kernel, as per tcpliveplay documentation. Kernel version is 5.4.0-kali2-amd64.

[fklassen]

I discussed with an early maintainer, and unless we find a volunteer to take on tcpliveplay, we will drop support in version 4.4.

[spgyip]

I have just downloaded and built the latest version from github and fail also with the other available injection methods:

tcpliveplay version: 4.3.2 (build git:v4.3.2)
Copyright 2012 by Yazan Siam <tcpliveplay@gmail.com>
Not compiled with libdnet.
Compiled against libpcap: 1.9.1
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Injection method: pcap_sendpacket()

and

tcpliveplay version: 4.3.2 (build git:v4.3.2)
Copyright 2012 by Yazan Siam <tcpliveplay@gmail.com>
Not compiled with libdnet.
Compiled against libpcap: 1.9.1
64 bit packet counters: enabled
Verbose printing via tcpdump: enabled
Injection method: pcap_inject()

Here is the output of one run:

root@kali:~/Downloads/tcpreplay-4.3.2# src/tcpliveplay eth0 ~/service-connection.pcap 192.168.0.5 re:dac:ted:6C:CB:BB random
new source port:: 59506
Random Local SEQ: 140580199
Packets Scheduled 241
Sending Local Packet...............     [1]
Receiving Packets from remote host...
Received Remote Packet...............   [2]
Remote Pakcet Expectation met.
Proceeding in replay....

======================================================================
= TIMEOUT:: Remote host is not responding. You may have crashed      =
= the host you replayed these packets against OR the packet sequence =
= changed since the capture was taken resulting in differing         =
= expectations. Closing replay...                                    =
======================================================================

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ Unfortunately an error has occurred  halting the replay of
~ the pcap file '/root/service-connection.pcap'. Please see error above for details...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

----------------TCP Live Play Summary----------------
- Packets Scheduled to be Sent & Received:          241
- Actual Packets Sent & Received:                   2
- Total Local Packet Re-Transmissions due to packet
- loss and/or differing payload size than expected: 0
- Thank you for Playing, Play again!
----------------------------------------------------------

What I can see in a parallel wireshark session is, that my machine is sending out a TCP SYN packet and the server is responding with a SYN ACK. No ACK is given by my machine. After a few seconds, the server resends the SYN ACK a couple of times. Finally the program aborts, claiming that the server did not respond. Opening a TCP session with telnet to the same port works just fine. There are no iptables rules, except for the one suppressing the RST from the kernel, as per tcpliveplay documentation. Kernel version is 5.4.0-kali2-amd64.

I am running on Centos8 with tcpliveplay v4.3.4, exactly the same problem as @fogs . I have tcpdump all the packets, and, it appears that the server side keeps replying with SYN ACK, but tcpliveplay just don't get the packet, and then TIMEOUT.

[angrybunny123]

Hi everyone, I am currently trying out tcpliveplay v4.4.2 and am experiencing the same issue with tcpliveplay not sending the ACK packet to the server after server sends SYN ACK. Wondering if this issue has been fixed or if there is any alternative tool to use? Thanks.

[jewdas1984]

Hello @fklassen - I've ran into the very same issue, is there any possibility to overcome it?

[tflament]

I have encountered the same issue and looking at the code this seems to be caused by function pcap_dispatch that is blocking for 10 seconds before returning :

pcap_dispatch(live_handle, 1, got_packet, NULL); /* Listen in on NIC for tcp packets */

This seems legitimate with the use of the packet buffer timeout value of 10000 ms on the pcap_open_live function :

#define TIMEOUT_ms 10000

/* Open the session in promiscuous mode */
handle = pcap_open_live(dev, BUFSIZ_PLUS, PROMISC_OFF, TIMEOUT_ms, errbuf);

So basically pcap_dispatch is trying to get as much packets as possible for 10 seconds and then return them in bulk.

Maybe the behavior was different on implementations that did not support this packet buffer timeout (for example more like a global timeout value for receiving any packet but that would have proceed any received packet immediately)

The issue is that timeout alarm signal is also set to be fired after 10 seconds, and you actually see the second packet being received by tcpliveplay just shortly before the timeout message is shown :

new source port:: 56760
Random Local SEQ: 1743999685
Packets Scheduled 9
Sending Local Packet............... [1]
Receiving Packets from remote host...
[around 10 seconds of delay]
Received Remote Packet...............   [2] <- reply packet well received by tcpliveplay, but only after 10 seconds
...

A quick test has been to reduce TIMEOUT_ms to something much lower, for example 200ms. Now tcpliveplay seems to process further and send ACK reply packet .

However in my test scenario there was an issue with the ACK value sent for this third packet, the value was incorrect. I don't know if this can be related to change I have made or if there was an issue in the ack value computed, I will have to further investigate the seq/ack computation to find out.

Also during my tests I have seen that on Linux implementation of libpcap pcap_dispatch never returns if no packet is received, this is because the packet buffer timeout only starts when the first packet is received.

Man page state that this is totally dependent on implementation and may differ, so we should not rely on this :

NOTE: the packet buffer timeout cannot be used to cause calls that read packets to return within a limited period of time, because, on some platforms, the packet buffer timeout isn't supported, and, on other platforms, the timer doesn't start until at least one packet arrives. This means that the packet buffer timeout should NOT be used, for example, in an interactive application to allow the packet capture loop to ``poll'' for user input periodically, as there's no guarantee that a call reading packets will return after the timeout expires even if no packets have arrived.

This also means that pcap_dispatch may on some implementation return a value of 0 after 200 ms (no packet received), which may cause issue if our remote packet is delayed.

So maybe a better approach would be to loop until some packets are received :

int cnt;
while ((cnt = pcap_dispatch(live_handle, 1, got_packet, NULL)) == 0 && keep_going);

And pcap_breakloop when timeout alarm is fired :

void
catch_alarm(int sig)
{
    keep_going = 0;

    /* If no packet captured, pcap_dispatch may never return depending on implementation, break the loop */
    pcap_breakloop(live_handle);

    signal(sig, catch_alarm);
}

Without this pcap_breakloop in catch_alarm if tcpliveplay see no packet pcap_dispatch may be blocking forever on implementations that do not start the packet buffer timeout before first packet is received, so we never see the timeout warning as we don't return to the main loop (the program just hangs forever)

I will try to investigate the second issue I had with wrong ACK value, and if positive can attempt to submit a PR, however I am not a C developer nor have I ever contributed to an open source project so I would be very happy for any of your feedback