search

google / gopacket

Provides packet processing capabilities for Go
BSD 3-Clause "New" or "Revised" License
6.26k stars 1.12k forks source link

very slow performance with pfring... #185

Open rudyeeee opened 8 years ago

rudyeeee commented 8 years ago

regardless of the number of slots, Using the pfring in 1gbps occured a lot of drop(Tot pkt Lost)

gconnell commented 8 years ago

Could you provide a little more detail on the code you're running? Do you have a minimum viable program that exhibits this?

If you're using PacketSource or ReadPacketData, maybe switch to ReadPacketDataTo to avoid an extra alloc. Doesn't look like we've implemented ZeroCopyReadPacketData for pfring, but we probably should.

Are you doing other stuff in your code, tcp assembly or the like?

Could you generate a CPU profile with your run (https://golang.org/pkg/runtime/pprof/) and generate a SVG of its output to add here? That might point to a smoking gun.

rahulpowar commented 7 years ago

I have a similar issue with PF_RING. Specifically the latencies seem very high.

I have created a minimal ping using gopacket and pfring here: https://github.com/redsift/pingdemo

When testing this on Ubuntu 16.04.1 LTS with PF_RING 6.6.0 and the pfring e1000e driver, the latency suggests something is wrong.

The linked project fires a ping packet and times the response.

Hitting 8.8.8.8 takes ~500ms

PF_RING ping timing demo: ping 8.8.8.8 from 10.60.0.175
ping time=500.586337ms
ping time=500.571822ms
ping time=500.571827ms
ping time=500.575797ms
ping time=500.577616ms

The same test with the zero copy interface is mostly the same though I guess this is not relevant for my single packet test case.

PF_RING ping timing demo: ping 8.8.8.8 from 10.60.0.175
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC license 6.6.0.170424 for eno1 [Intel 1 Gbit e1000e family]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and
# transmission limited to 5 minutes
#########################################################################
ping time=500.57761ms
ping time=500.579852ms
ping time=500.565223ms
ping time=500.56318ms
ping time=500.569313ms

As a control, command line ping reports tiny latencies.

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=4.50 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=4.12 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=4.07 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=59 time=4.41 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=59 time=4.20 ms

In another application, gopacket with libpcap reports times within 10% of the command line ping to this ~500ms latency is specific to PF_RING.

What am I missing?

gconnell commented 7 years ago

Not sure if this is correct, but my guess is that PF_RING might be trying to buffer multiple packets before returning the set to you, the caller. If you're only seeing one, it may be waiting for a timeout (500ms) before returning. I'll dig briefly into the code and see if that makes sense.

gconnell commented 7 years ago

Hmm, there's a very suspicious pfring global called DEFAULT_POLL_DURATION set to 500.

gconnell commented 7 years ago

We might need to expose pfring_set_poll_duration to fix this...

gconnell commented 7 years ago

Could you throw some printf-debugging around the ReadPacketDataTo call and make sure it's the function that's actually taking ~500ms?

rahulpowar commented 7 years ago

Yes, the 500ms is in ReadPacketDataTo.

Looks like you are right. Combination of pfring_set_poll_duration (default = 500) and pfring_set_poll_watermark (default = 128). Clients of gopacket likely want to tweak these based on use case.

rahulpowar commented 6 years ago

I suspect the original issue here is not what I was reporting but looks like https://github.com/google/gopacket/issues/185 clears my problem.

gkodali-zededa commented 6 years ago

I have added SetPollDuration, SetPollWatermark APIs for performance tweaking needs. Pulled to master recently.

gujunyan commented 5 years ago

PF_RING NOT FREE. I got same problem . i use dpdk at last.

nicolas-b12 commented 5 years ago

Same problem here, in input flow. Sending a UDP packet with netcat, netcat exits almost immediately but I see the packet only 2 or 3 seconds later. The ping to the remote is 20-30 ms