search

seladb / PcapPlusPlus

PcapPlusPlus is a multiplatform C++ library for capturing, parsing and crafting of network packets. It is designed to be efficient, powerful and easy to use. It provides C++ wrappers for the most popular packet processing engines such as libpcap, Npcap, WinPcap, DPDK, AF_XDP and PF_RING.
https://pcapplusplus.github.io/
The Unlicense
2.68k stars 648 forks source link

Forwarding Packets causes bad hdr length 40 - too long #497

Closed gerald-dotcom closed 4 years ago

gerald-dotcom commented 4 years ago

I'm using PF_Ring to forward incoming packets (WITHOUT A SINGLE MODIFICATION) to a Bridge and running TCP Dump inside Bridge display a lot of logs that says bad hdr length 40 - too long.

My server is using virtio network interface and appear to be a KVM VM, bought from linode because VPS is cheaper for me than bare-metal.

DPDK is not the solution, don't direct me into that one. It might be easy for you but for me isn't.

My code,

#include <iostream>
#include <string>
#include <array>
#include <vector>
#include <map>
#include <chrono>
#include <time.h>
#include <cstdint>
#include <sstream>
#include <iomanip>
#include <stdexcept>
#include <PfRingDevice.h>
#include <PfRingDeviceList.h>
#include <Packet.h>
#include <EthLayer.h>
#include <IPv4Layer.h>
#include <PacketUtils.h>
#include <IPv6Layer.h>
#include <TcpLayer.h>
#include <SSLLayer.h>
#include "dataObjects.h"
#include <PcapPlusPlusVersion.h>

using namespace pcpp;

void packetArrived(RawPacket* packets, std::uint32_t numOfPackets, std::uint8_t threadId, PfRingDevice* device, void* userCookie) {
    for (unsigned int i = 0; i < numOfPackets; i++) { 

        packetFingerprint *dObj = (packetFingerprint*)userCookie;

        auto outgoing = dObj->sendPacketsTo;

        // Parse Current Packet
        pcpp::Packet parsedPacket(&packets[i]);

        // Send It Out

        outgoing->sendPacket(parsedPacket);

    }
}

int main(){

    std::string defaultInterface = "eth0";

    // Get Instance of Interface
    auto device = pcpp::PfRingDeviceList::getInstance().getPfRingDeviceByName(defaultInterface.c_str());

    if (device == NULL)
    {
        std::cout << "Couldn't locate default Network Driver"
                  << "\n";
        return 1;
    }

    int systemCores = pcpp::getNumOfCores(); // This will usually 2 
    int numOfCaptureThreads = systemCores-1; // This will set 2-1 = 1 

    pcpp::CoreMask cores = 0;

    int threadId = 0;
    int threadCount = 0;

    while (threadCount < numOfCaptureThreads)
    {
        if (SystemCores::IdToSystemCore[threadId].Id != device->getCurrentCoreId().Id)
        {
            cores |= SystemCores::IdToSystemCore[0].Mask;
            threadCount++;
        }

        threadId++;
    }

    if (!device->openMultiRxChannels(numOfCaptureThreads, PfRingDevice::PerFlow)) {
        std::cout << "Couldn't open RX Channels"
                  << "\n";
        return 1;
    }

    // Initialize Data Object

    packetFingerprint dObj;

    // Outgoing Interface (Bridge)
    std::string outgoingInterface = "br0";

    // Initialize and Set Outgoing Interface on Data Object

    dObj.sendPacketsTo = pcpp::PfRingDeviceList::getInstance().getPfRingDeviceByName(outgoingInterface.c_str());

    // Open Outgoing Interface and make it ready to accept packets

    if(!dObj.sendPacketsTo->open()) {
        std::cout << "Unable to open Outgoing Interface" << "\n";
    }

    // Start Packet Capturing

    std::cout << "Started Packet Capturing"
              << "\n";

    if(!device->startCaptureMultiThread(packetArrived, &dObj, cores)) {
        std::cout << "Unable To Start Packet Capturing" << "\n";
    }

}

You can see full log below,

23:53:18.751094 IP (tos 0x28, ttl 51, id 50222, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0xaa43 (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185016705 ecr 0,nop,wscale 7], length 0
23:53:18.751605 IP (tos 0x0, ttl 64, id 39363, offset 0, flags [DF], proto UDP (17), length 73)
    192.168.1.11.37078 > resolver04.tor1.linode.com.domain: 16733+ PTR? 129.25.105.172.in-addr.arpa. (45)
23:53:18.752494 IP (tos 0x0, ttl 62, id 22840, offset 0, flags [none], proto UDP (17), length 206)
    resolver04.tor1.linode.com.domain > 192.168.1.11.37078: 16733 1/5/0 129.25.105.172.in-addr.arpa. PTR li1980-129.members.linode.com. (178)
23:53:18.752579 IP (tos 0x0, ttl 64, id 27582, offset 0, flags [DF], proto UDP (17), length 72)
    192.168.1.11.52523 > resolver09.tor1.linode.com.domain: 27743+ PTR? 252.42.143.63.in-addr.arpa. (44)
23:53:18.753459 IP (tos 0x0, ttl 62, id 24971, offset 0, flags [none], proto UDP (17), length 110)
    resolver09.tor1.linode.com.domain > 192.168.1.11.52523: 27743 1/0/0 252.42.143.63.in-addr.arpa. PTR engine27.uptimerobot.com. (82)
23:53:18.753525 IP (tos 0x0, ttl 64, id 39364, offset 0, flags [DF], proto UDP (17), length 70)
    192.168.1.11.58628 > resolver04.tor1.linode.com.domain: 60441+ PTR? 5.5.105.172.in-addr.arpa. (42)
23:53:18.754075 IP (tos 0x0, ttl 62, id 22841, offset 0, flags [none], proto UDP (17), length 200)
    resolver04.tor1.linode.com.domain > 192.168.1.11.58628: 60441 1/5/0 5.5.105.172.in-addr.arpa. PTR resolver04.tor1.linode.com. (172)
23:53:18.754161 IP (tos 0x0, ttl 64, id 39365, offset 0, flags [DF], proto UDP (17), length 71)
    192.168.1.11.53102 > resolver04.tor1.linode.com.domain: 18679+ PTR? 11.1.168.192.in-addr.arpa. (43)
23:53:18.755245 IP (tos 0x0, ttl 64, id 27583, offset 0, flags [DF], proto UDP (17), length 70)
    192.168.1.11.60272 > resolver09.tor1.linode.com.domain: 49297+ PTR? 5.7.105.172.in-addr.arpa. (42)
23:53:18.755932 IP (tos 0x0, ttl 62, id 24972, offset 0, flags [none], proto UDP (17), length 200)
    resolver09.tor1.linode.com.domain > 192.168.1.11.60272: 49297 1/5/0 5.7.105.172.in-addr.arpa. PTR resolver09.tor1.linode.com. (172)
23:53:19.739695 IP (tos 0x28, ttl 51, id 50223, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0xa64a (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185017722 ecr 0,nop,wscale 7], length 0
23:53:21.436565 IP (tos 0x2,ECT(0), ttl 120, id 62002, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62719 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:21.436748 IP (tos 0x2,ECT(0), ttl 120, id 62003, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62720 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:21.436901 IP (tos 0x0, ttl 64, id 39489, offset 0, flags [DF], proto UDP (17), length 73)
    192.168.1.11.40008 > resolver04.tor1.linode.com.domain: 42094+ PTR? 194.19.171.192.in-addr.arpa. (45)
23:53:21.437327 IP (tos 0x0, ttl 62, id 23237, offset 0, flags [none], proto UDP (17), length 132)
    resolver04.tor1.linode.com.domain > 192.168.1.11.40008: 42094 NXDomain 0/1/0 (104)
23:53:21.690868 IP (tos 0x2a,ECT(0), ttl 115, id 62005, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62721 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:21.787506 IP (tos 0x28, ttl 51, id 50224, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0x9e4a (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185019770 ecr 0,nop,wscale 7], length 0
23:53:23.762337 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.10 tell 192.168.1.11, length 28
23:53:23.762385 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.10 is-at ce:19:71:de:38:eb (oui Unknown), length 28
23:53:23.762501 IP (tos 0x0, ttl 64, id 39899, offset 0, flags [DF], proto UDP (17), length 71)
    192.168.1.11.60792 > resolver04.tor1.linode.com.domain: 10490+ PTR? 10.1.168.192.in-addr.arpa. (43)
23:53:23.763311 IP (tos 0x0, ttl 62, id 23881, offset 0, flags [none], proto UDP (17), length 130)
    resolver04.tor1.linode.com.domain > 192.168.1.11.60792: 10490 NXDomain* 0/1/0 (102)
23:53:24.438261 IP (tos 0x2,ECT(0), ttl 120, id 62008, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62719 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:24.438275 IP (tos 0x2,ECT(0), ttl 120, id 62007, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62720 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:24.691701 IP (tos 0x2a,ECT(0), ttl 115, id 62010, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62721 > li1980-129.members.linode.com.https: Flags [SEW] [bad hdr length 40 - too long, > 32]
23:53:25.831118 IP (tos 0x28, ttl 51, id 50225, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0x8e8a (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185023802 ecr 0,nop,wscale 7], length 0
23:53:28.770323 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.11 tell 192.168.1.10, length 28
23:53:28.770331 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.11 is-at a2:82:95:08:1c:7f (oui Unknown), length 28
23:53:30.439105 IP (tos 0x0, ttl 120, id 62013, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62720 > li1980-129.members.linode.com.https: Flags [S] [bad hdr length 40 - too long, > 32]
23:53:30.439131 IP (tos 0x0, ttl 120, id 62012, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62719 > li1980-129.members.linode.com.https: Flags [S] [bad hdr length 40 - too long, > 32]
23:53:30.697469 IP (tos 0x28, ttl 115, id 62015, offset 0, flags [DF], proto TCP (6), length 52)
    THIS_IS_MY_HOME_IP_HIDDEN_PRIVACY.62721 > li1980-129.members.linode.com.https: Flags [S] [bad hdr length 40 - too long, > 32]
23:53:34.081105 IP (tos 0x28, ttl 51, id 50226, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0x6e4a (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185032058 ecr 0,nop,wscale 7], length 0
23:53:50.499259 IP (tos 0x28, ttl 51, id 50227, offset 0, flags [DF], proto TCP (6), length 60)
    engine27.uptimerobot.com.47974 > li1980-129.members.linode.com.https: Flags [S], cksum 0x2e4a (correct), seq 3613604393, win 29200, options [mss 1460,sackOK,TS val 185048442 ecr 0,nop,wscale 7], length 0
gerald-dotcom commented 4 years ago

Just to let you know, I tried this on Bare Metal instance and it didn't worked either. Same output!!

gerald-dotcom commented 4 years ago

Well, I had chance to get the packet and frame lengths so here is the output of both of them,

Raw Packet Size is 66 Raw Frame Length is 66

seladb commented 4 years ago

I'm not sure why this error is happening. It's coming from tcpdump, right? You probably need to debug it on your own environment. I don't think I can reproduce it on my environment.

gerald-dotcom commented 4 years ago

I'm not sure why this error is happening. It's coming from tcpdump, right? You probably need to debug it on your own environment. I don't think I can reproduce it on my environment.

Yes, It's coming from tcpdump.

I believe a bug reside in your lib because I used another lib called libtins and it forwarded packets, TCP Dump didn't show any error for that lib.

You can create a Docker container and run TCPDump inside it then forward packets to docker0 interface.

gerald-dotcom commented 4 years ago

@seladb I tried your Pcap implementation and it's packet sending function, and can confirm, your pcap implementation works and shows correct output in TcpDump so I believe the bug resides in your PF_RIng's implementation

seladb commented 4 years ago

Yes that makes sense. Can you try to debug the issue on your environment? Let me know if you find the bug.

BTW, libtins uses libpcap under the hood so it makes sense that both libtins and PcapPlusPlus work with the libpcap implementation. So the question is what's wrong with the PF_RING implementation. It's possible that the issue resides on the PF_RING side also. You need to further debug.

gerald-dotcom commented 4 years ago

I'm unable debug your lib due to the complexity.

From: seladb notifications@github.com Sent: Saturday, July 11, 2020 6:42 PM To: seladb/PcapPlusPlus PcapPlusPlus@noreply.github.com Cc: Cornelius Jackson cornelius@neliusproperties.com; Author author@noreply.github.com Subject: Re: [seladb/PcapPlusPlus] Forwarding Packets causes bad hdr length 40 - too long (#497)

Yes that makes sense. Can you try to debug the issue on your environment? Let me know if you find the bug

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/seladb/PcapPlusPlus/issues/497#issuecomment-657148135, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOQKQCRYJO5BJSZ4IK7IYA3R3D2FZANCNFSM4OVCHHUA.

seladb commented 4 years ago

I don't think debugging it should be very complex. All you need to debug is PfRingDevice, you can use gdb or add printf statements.

gerald-dotcom commented 4 years ago

@seladb I don't think debugging would solve this. This has to do with how you parse packets because I see 60 total length of each packet.

Sorry, I don't have time to fiddle with this.

I will go with pcap.