Dropping Packets Error Code 6?

[MrRandom92]

Hello Jesse, I am trying to use Divert.Net to create a simple packet latency simulator, could you explain why the packets are being randomly dropped?

    Thread receiving;
    Thread sending;
    Diversion diversion;

    public struct packet
    {
        public Address address;
        public byte[] buffer;
        public uint receiveLength;
        public uint sendLength;
        public DateTime dt;
    }

    packet rp = new packet();
    packet sp = new packet();
    Queue pq = new Queue();

    public int pc = 0; //packet count

    public void run()
    {
        rp.address = new Address();
        rp.buffer = new byte[65535];
        rp.receiveLength = 0;
        rp.sendLength = 0;
        rp.dt = DateTime.Now;

        while (true)
        {
            if (!diversion.Receive(rp.buffer, rp.address, ref rp.receiveLength))
            {
                 continue;
            }
            else
            {
                rp.dt = DateTime.Now.AddTicks(1000000); //adds timestamp + 100ms
                pq.Enqueue(rp); //adds packet to queue
                pc++;
           }
       }
    }

    public void send()
    {
        while (true)
        {
            if (pc > 0)
            {
                sp = (packet)pq.Peek(); //gets first packet in the queue
                if (sp.dt <= DateTime.Now) //checks if packet arrival time is <= than current time
                {
                    //diversion.SendAsync(sp.buffer, sp.receiveLength, sp.address, ref sp.sendLength);
                    if (!diversion.Send(sp.buffer, sp.receiveLength, sp.address, ref sp.sendLength))
                    {
                        try
                        {
                            Invoke(new Action(() => { richTextBox1.AppendText(System.Runtime.InteropServices.Marshal.GetLastWin32Error().ToString() + "\n"); }));
                        }
                        catch { }
                        pq.Dequeue();
                        pc--;
                    }
                    else
                    {
                        pq.Dequeue();
                        pc--;
                    }
                }
            }
        }
    }

    private void button1_Click(object sender, EventArgs e)
    {
        diversion = Diversion.Open("outbound", DivertLayer.Network, 100, 0);

        receiving = new Thread(run);
        receiving.Start();

        sending = new Thread(send);
        sending.Start();
    }

    private void button2_Click(object sender, EventArgs e)
    {
        sending.Abort();
        receiving.Abort();
        if(pq.Count > 0)
        {
            pq.Clear();
        }
        diversion.Close();
    }`

This is using Windows Forms and I have compiled Divert.Net to x64.

Side note: I am a beginner with this, but if you could advise it would be appreciated.

[TechnikEmpire]

@TodM0 Where are you getting error code 6? If you look here, and you're properly fetching the error code, then error 6 means ERROR_INVALID_HANDLE which means that you're performing an operation against an invalid handle.

It's been a while, but there should be something like Valid or IsValid property on the diversion object to ensure that you've got a valid handle. Ensure that you're running as administrator, and possibly try using the PInvoke version at https://github.com/TechnikEmpire/DivertPInvoke. This way you can follow the official documentation exactly and greatly eliminate the odds that a bug is due to my wrapper. It's also less annoying because you can stick to AnyCPU targets.

[MrRandom92]

@TechnikEmpire Thanks for the quick reply. I am getting the error after diversion.send, but if I remove the timestamp checks, so that there is no delay in sending the packets everything works fine. I did try the PInvoke version prior to Divert.Net, and both produce the same random dropped packets when trying to delay them from sending instantly.

[TechnikEmpire]

If I had to guess, I'd say that you're overflowing the queue in WinDivert by waiting too long. There is one and you can configure it to be larger (search the docs) and that might help, but what I would suggest is switching to a threaded implementation using both the asynchronous send and receive functions. This way at least you'll be parallelizing the work so your stacking up of packets on the internal queue isn't linear and therefore exponential.

That's of course all assuming that my guess is accurate as to the cause.

On Wed, Jan 25, 2017 at 1:22 AM TodM notifications@github.com wrote:

@TechnikEmpire https://github.com/TechnikEmpire Thanks for the quick reply. I am getting the error after diversion.send, but if I remove the timestamp checks, so that there is no delay in sending the packets everything works fine. I did try the PInvoke version prior to Divert.Net, and both produce the same random dropped packets when trying to delay them from sending instantly.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/TechnikEmpire/Divert.Net/issues/12#issuecomment-275029233, or mute the thread https://github.com/notifications/unsubscribe-auth/AKtty6SV51fiXrzDaU6-rYiV51VGqzrMks5rVupCgaJpZM4LtGY- .

-- Jesse Nicholson

[MrRandom92]

@TechnikEmpire I have tried increasing the WinDivert queue length to the maximum with no luck. Should the WinDivert queue length really be an issue when WinDivert automatically drops packets after a set queue time anyway? If the packets are sent by my own C# queue I fail to understand why they wouldn't be being sent all of the time, even though the packets are sent the majority of the time with the added 100ms delay as required. I have also ensured my own queue is not becoming full.

I will delve into the asynchronous methods, but I did not expect it to be this difficult to simply add delay to packets. Looking at my code provided can you see any fault or provide any pointers on how to change it to be asynchronous? Thanks for the support.

[TechnikEmpire]

I'll do an actual test myself just give me a few. I was taking a stab in the dark but you have valid objections.

[MrRandom92]

@TechnikEmpire Thanks. Using ping google.com -t is how I am seeing the lost packets.

[TechnikEmpire]

@TodM0 My best guess now is that you're not calling the CalculateChecksums functions. It's a method of the Diversion class in this library. Below is a threaded example using the PInvoke library that functions exactly as you intend with zero packet loss.

using System;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace TestDivert
{
    internal unsafe class Program
    {
        public static volatile bool running = true;

        private static void Main(string[] args)
        {
            Console.CancelKeyPress += delegate
            {
                running = false;
            };

            var divertHandle = WinDivertMethods.WinDivertOpen("outbound", WINDIVERT_LAYER.WINDIVERT_LAYER_NETWORK);

            if(divertHandle != IntPtr.Zero)
            {
                Parallel.ForEach(Enumerable.Range(0, Environment.ProcessorCount), x => RunDiversion(divertHandle));
            }

            WinDivertMethods.WinDivertClose(divertHandle);
        }

        private static void RunDiversion(IntPtr handle)
        {
            Console.WriteLine("Starting diversion thread.");

            byte[] packet = new byte[65535];

            while(running)
            {
                WINDIVERT_IPHDR* ipv4Header = null;
                WINDIVERT_IPV6HDR* ipv6Header = null;
                WINDIVERT_ICMPHDR* icmpv4Header = null;
                WINDIVERT_ICMPV6HDR* icmpv6Header = null;
                WINDIVERT_TCPHDR* tcpHdr = null;
                WINDIVERT_UDPHDR* udpHdr = null;

                uint readLength = 0;
                WINDIVERT_ADDRESS addr = new WINDIVERT_ADDRESS();
                if(!WinDivertMethods.WinDivertRecv(handle, packet, (uint)packet.Length, ref addr, ref readLength))
                {
                    continue;
                }

                ////Console.WriteLine("Read {0} bytes into packet, packet direction is {1}.", readLength, addr.Direction.ToString());
                ////Console.WriteLine(BitConverter.ToString(packet, 0, (int)readLength));

                fixed (byte* inBuf = packet)
                {
                    byte* payload = null;

                    uint payloadLen = 0;

                    WinDivertMethods.WinDivertHelperParsePacket(inBuf, readLength, &ipv4Header, &ipv6Header, &icmpv4Header, &icmpv6Header, &tcpHdr, &udpHdr, &payload, &payloadLen);

                    if(ipv4Header != null && tcpHdr != null && payload != null)
                    {
                        // Do IPV4 TCP specific stuff here where a payload is present.

                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP IPV6 packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }

                    if(tcpHdr != null && ipv4Header != null)
                    {
                        // Do IPV4 TCP specific stuff here where no payload is present.

                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }
                    else if(tcpHdr != null && ipv6Header != null)
                    {
                        // Do IPV6 TCP specific stuff here.

                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;

                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP IPV6 packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }

                    if(ipv4Header != null && udpHdr != null)
                    {
                        // Do IPV4 UDP specific stuff here.

                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;

                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got UDP packet from port {0} to port {1}.", udpHdr->SrcPort, udpHdr->DstPort);
                    }
                }

                WinDivertMethods.WinDivertHelperCalcChecksums(packet, readLength, 0);

                // Halt the sending of this packet for roughly 100 msec.
                Thread.Sleep(100);

                WinDivertMethods.WinDivertSendEx(handle, packet, readLength, 0, ref addr, IntPtr.Zero, IntPtr.Zero);
            }
        }
    }
}

[TechnikEmpire]

[MrRandom92]

@TechnikEmpire The example code is great, and reduces the loss a lot, but I am still seeing lost packets unfortunately when left to run for longer.

Is this expected behaviour?

[TechnikEmpire]

@TodM0 I'm running with 500 msec delays with zero loss right now. I don't think long running pings are a very reliable test for real network issues. If it's anything, I'd argue that it's not the diversion library. I'm on a flaky network myself (LTE connection) and I'm seeing zero loss.

[TechnikEmpire]

@TodM0 If it's any reassurance, I'm using WinDivert in a commercial-grade content filter that does inline HTTP/S deep content inspection and classification, as well as blocking based on that analysis. It's very stable.

[MrRandom92]

@TechnikEmpire I am just using the regular WinDivert, the pings are also randomly climbing, I know ping tests aren't the best way, but this shouldn't be happening. Does this give you any further clues as to what it could be?

If not I will assume it's a fault somewhere on my machine and will post back when I do some further research and tests. What I do know is applications such as Clumsy which use WinDivert work flawlessly on my machine to increase ping.

[TechnikEmpire]

@TodM0 I'm not sure, this code is a direct pinvoke into WinDivert so it's not anything my code is doing. I'd look at handle use and priority on Clumsy and compare. Also if you're running clumsy maybe there's two instances of the divert driver running now. You can try 'sc stop WinDivert1.2" and same for WinDiver1.1 or whatever version clumsy has to ensure it's not a driver conflict. Lastly you could remove any firewall software and do Netsh winsock reset and reboot then re-test.

[MrRandom92]

@TechnikEmpire I am a little confused how using Thread.Sleep() doesn't cause the queue to become blocked up. Wouldn't newly arriving packets during the sleep time become stalled too, possibly mounting up the delay? For example: Packet 2 arrives -> Packet 1 Arrives -> 100ms -> Send Packet. Packet 2 may now be waiting in excess of 150ms before being sent, or does WinDivert take care of this somehow?

When increasing the queue time and queue length of WinDivert to their max the packets are delayed for up to 2000+ms, when Thread.Sleep is only at 100ms, before the request times out, and it's also reducing my bandwidth from 30Mbps to 2Mbps.

`#define WINDIVERT_PARAM_QUEUE_LEN_DEFAULT 1024

define WINDIVERT_PARAM_QUEUE_LEN_MIN 1

define WINDIVERT_PARAM_QUEUE_LEN_MAX 8192

define WINDIVERT_PARAM_QUEUE_TIME_DEFAULT 512

define WINDIVERT_PARAM_QUEUE_TIME_MIN 128

define WINDIVERT_PARAM_QUEUE_TIME_MAX 2048`

Removing Thread.Sleep() and starting a speedtest causes high ping spikes suggesting the problem is with the WinDivert queue or the Pinvoke wrapper itself.

[TechnikEmpire]

@TodM0 Regarding the buffer not overflowing, I'd say because such protocols are not going to just be blindly pumping out a nonstop stream of packets. They're sending, then waiting for a response.

With regards to the params, are you setting them correctly, or are you redefining header defs?

https://reqrypt.org/windivert-doc-v1.2.html#divert_set_param

Here's an example in code of me setting those params (c++): https://github.com/TechnikEmpire/HttpFilteringEngine/blob/master/src/te/httpengine/mitm/diversion/impl/win/WinDiverter.cpp#L122

I have no problems with the WinDivert queue personally so I can't relate or comment on those. If you're starting to have WinDivert specific issues that you truly think are bugs, then please post on basil's repo (official WinDivert repo).

[MrRandom92]

@TechnikEmpire It seems that packets are having to wait too long using Thread.Sleep() when performing bandwidth intensive events causing the ping to spike and occasionally lose packets.

I have tried modifying your example code to copy the packets into a new queue with timestamps to compensate for newly arriving packets during the active delay.

My problem now is that packets are refusing to send with error code 87 (invalid parameter), but the filter is correct it. This error is thrown upon attempting to send a packet after a set delay from the new queue.

Example:

using System;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Collections.Generic;

namespace DivertFix
{
    internal unsafe class Program
    {
        public static volatile bool running = true;
        public static Stopwatch sw = new Stopwatch();

        public struct packet
        {
            public byte[] data;
            public uint readL;
            public WinDivert.WINDIVERT_ADDRESS addr;
            public long timestamp;
        }

        public static Queue<packet> packet_queue = new Queue<packet>();

        private static void Main(string[] args)
        {
            Console.CancelKeyPress += delegate
            {
                running = false;
                sw.Stop();
            };

            var divertHandle = WinDivert.WinDivertMethods.WinDivertOpen("outbound", WinDivert.WINDIVERT_LAYER.WINDIVERT_LAYER_NETWORK, 1000, 0);

            if (divertHandle != IntPtr.Zero)
            {
                Parallel.ForEach(Enumerable.Range(0, Environment.ProcessorCount), x => RunDiversion(divertHandle));
            }

            WinDivert.WinDivertMethods.WinDivertClose(divertHandle);
        }

        private static void RunDiversion(IntPtr handle)
        {
            Console.WriteLine("Starting diversion thread.");

            byte[] packet = new byte[65535];
            sw.Start();

            while (running)
            {

                WinDivert.WINDIVERT_IPHDR* ipv4Header = null;
                WinDivert.WINDIVERT_IPV6HDR* ipv6Header = null;
                WinDivert.WINDIVERT_ICMPHDR* icmpv4Header = null;
                WinDivert.WINDIVERT_ICMPV6HDR* icmpv6Header = null;
                WinDivert.WINDIVERT_TCPHDR* tcpHdr = null;
                WinDivert.WINDIVERT_UDPHDR* udpHdr = null;

                uint readLength = 0;
                WinDivert.WINDIVERT_ADDRESS addr = new WinDivert.WINDIVERT_ADDRESS();
                if (!WinDivert.WinDivertMethods.WinDivertRecv(handle, packet, (uint)packet.Length, ref addr, ref readLength))
                {
                    continue;
                }

                ////Console.WriteLine("Read {0} bytes into packet, packet direction is {1}.", readLength, addr.Direction.ToString());
                ////Console.WriteLine(BitConverter.ToString(packet, 0, (int)readLength));
                fixed (byte* inBuf = packet)
                {
                    byte* payload = null;

                    uint payloadLen = 0;

                    WinDivert.WinDivertMethods.WinDivertHelperParsePacket(inBuf, readLength, &ipv4Header, &ipv6Header, &icmpv4Header, &icmpv6Header, &tcpHdr, &udpHdr, &payload, &payloadLen);

                    if (ipv4Header != null && tcpHdr != null && payload != null)
                    {
                        // Do IPV4 TCP specific stuff here where a payload is present.
                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP IPV6 packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }

                    if (tcpHdr != null && ipv4Header != null)
                    {
                        // Do IPV4 TCP specific stuff here where no payload is present.

                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }
                    else if (tcpHdr != null && ipv6Header != null)
                    {
                        // Do IPV6 TCP specific stuff here.
                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got TCP IPV6 packet from port {0} to port {1}.", tcpHdr->SrcPort, tcpHdr->DstPort);
                    }

                    if (ipv4Header != null && udpHdr != null)
                    {
                        // Do IPV4 UDP specific stuff here.
                        //var srcIp = ipv4Header->SrcAddr;
                        //var dstIp = ipv4Header->DstAddr;
                        //Console.WriteLine("From {0} to {1}", srcIp.ToString(), dstIp.ToString());
                        //Console.WriteLine("Got UDP packet from port {0} to port {1}.", udpHdr->SrcPort, udpHdr->DstPort);
                    }
                }

                WinDivert.WinDivertMethods.WinDivertHelperCalcChecksums(packet, readLength, 0);

                //Copy packet to the new queue
                packet newpacket = new packet();
                newpacket.data = packet;
                newpacket.readL = readLength;
                newpacket.addr = addr;
                newpacket.timestamp = sw.ElapsedMilliseconds + 100;
                packet_queue.Enqueue(newpacket);

                //Free(packet); Pop the WinDivert packet from buffer, not sure how or if required?

                //Get the first packet in the new queue
                packet getpacket = packet_queue.Peek();

                //Check if the current time is greater than packet arrival time
                if (sw.ElapsedMilliseconds >= getpacket.timestamp) //removing this if statement allows unlagged packets to send without issue
                {
                    WinDivert.WinDivertMethods.WinDivertSendEx(handle, getpacket.data, getpacket.readL, 0, ref getpacket.addr, IntPtr.Zero, IntPtr.Zero);
                    packet_queue.Dequeue(); //remove first packet from the queue
                    Console.WriteLine(System.Runtime.InteropServices.Marshal.GetLastWin32Error()); //Error 87 - possibly due to addr ref? 
                }
            }
        }
    }
}

Could it be that packets need dereferencing from the original queue before being put into the new queue? If you could take a quick look at the code provided, and removing the IF statement as commented you will see the code working, but without delay.

Thanks again for your time.

[TechnikEmpire]

@TodM0 while I wish I could be of further assistance, this thread has turned into general programming and networking questions. Please direct such questions to stackoverflow.com. I have over a dozen respositories with plenty of unsolved tickets in each of them and while I try to be accommodating, I don't have the availability to address this any further. I'd like to keep my bug tickets reserved for issues with the code. Thanks and all the best.

[MrRandom92]

@TechnikEmpire No problem and thanks for all the help. I have moved this issue over to stackoverflow if you ever get chance to take a further look at it here.