search

aregm / nff-go

NFF-Go -Network Function Framework for GO (former YANFF)
BSD 3-Clause "New" or "Revised" License
1.38k stars 156 forks source link

Any operation to free used memory in Mempool ? #687

Closed cang233 closed 4 years ago

cang233 commented 4 years ago

I write a program to capture the packets and build to flow using 5 tuple.I save the raw bytes of capturing packets and rebuild to pcaket.Packet struct after that. I run the program one night,and the log reminds me that the Mempool has less free space.And the slow operation has stopped in 7875 and no longer changged.

DEBUG: ---------------

DEBUG: System is using 4 cores now. 0 cores are left available.
DEBUG: Current speed of 0 instance of segment1 is 920322 PKT/S, cloneNumber: 1 queue number: 8
DEBUG: Current speed of 1 instance of segment1 is 847226 PKT/S, cloneNumber: 1 queue number: 8
DEBUG: Mempool usage receive 787 from 8191
DEBUG: Mempool usage receive1 958 from 8191
DEBUG: Mempool usage receive2 624 from 8191
DEBUG: Mempool usage receive3 651 from 8191
DEBUG: Mempool usage receive4 675 from 8191
DEBUG: Mempool usage receive5 553 from 8191
DEBUG: Mempool usage receive6 802 from 8191
DEBUG: Mempool usage receive7 677 from 8191
DEBUG: Mempool usage receive8 99 from 8191
DEBUG: Mempool usage receive9 101 from 8191
DEBUG: Mempool usage receive10 115 from 8191
DEBUG: Mempool usage receive11 105 from 8191
DEBUG: Mempool usage receive12 124 from 8191
DEBUG: Mempool usage receive13 101 from 8191
DEBUG: Mempool usage receive14 113 from 8191
DEBUG: Mempool usage receive15 127 from 8191
DEBUG: Mempool usage slow operations 7875 from 8191
DROP: slow operations mempool has less than 10% free space. This can lead to dropping packets while receive.
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105

I wanna ask that what's the reason of runing out of the Mempool memory? All I use the nff-go building struct is packet.Packet by NewPacket() function.Is there any method to release the memory I used? How can I effectively use the Mempool and release in-time to keep my program running?

cang233 commented 4 years ago

When I use the NewPacket() function only one time,the Mempool usage slow operations always 1.Seems that the NewPacket() function cause using out of the Mempool.And how can I release the new packet? Or how can I reuse only one packet object ?

gshimansky commented 4 years ago

To release memory your packet should reach a terminating point in a flow graph, it has to be either be sent, dropped or written to a file. Memory is released automatically in all of these conditions.

cang233 commented 4 years ago

But my operations is jumped out of the graph,now I only operate the single packet and can not release it.If there are any ways to put the packet generated by myself to a flow? Or I can release the single packet memory by myself ? I tried the SendPacket() to release it but meet a error:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7aad97]

runtime stack:
runtime.throw(0xab9c4a, 0x2a)
    /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 301 [syscall]:
runtime.cgocall(0x7ae2a0, 0xc000066e10, 0x0)
    /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000066de0 sp=0xc000066da8 pc=0x438ffb
github.com/intel-go/nff-go/internal/low._Cfunc_directSend(0x16a633e00, 0x16a630001, 0x0)
    _cgo_gotypes.go:500 +0x4a fp=0xc000066e10 sp=0xc000066de0 pc=0x5b704a
github.com/intel-go/nff-go/internal/low.DirectSend.func1(0x16a633e00, 0x1, 0x16a633f00)
    /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/internal/low/low.go:48 +0x60 fp=0xc000066e48 sp=0xc000066e10 pc=0x5ba150
github.com/intel-go/nff-go/internal/low.DirectSend(0x16a633e00, 0x1, 0x0)
    /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/internal/low/low.go:48 +0x35 fp=0xc000066e70 sp=0xc000066e48 pc=0x5b7ff5
github.com/intel-go/nff-go/packet.(*Packet).SendPacket(...)
    /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/packet/packet.go:848
flowparser/flows.save2file(0xc000b854a0)
    /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter_save_handler.go:119 +0x235 fp=0xc000066f60 sp=0xc000066e70 pc=0x5c1825
flowparser/flows.saveSamples(0xc000b854a0)
    /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter_save_handler.go:72 +0xe3 fp=0xc000066fb0 sp=0xc000066f60 pc=0x5c15d3
flowparser/flows.init.2.func1.3(0xc0000140d0, 0xc000b854a0)
    /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter.go:49 +0x2b fp=0xc000066fd0 sp=0xc000066fb0 pc=0x5c2afb
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000066fd8 sp=0xc000066fd0 pc=0x490d71
created by flowparser/flows.init.2.func1
    /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter.go:48 +0xf7

Here is part of my code:

    newp, err := packet.NewPacket()
    if err != nil {
        common.LogDebug(common.Debug, "New packet Error:,", err.Error())
        return
    }
        packet.GeneratePacketFromByte(newp, v.RawBytes)
        if !success {
            common.LogDebug(common.Debug, "Can not generate packet from byte.")
            return
        }
        newp.WritePcapOnePacket(file)
        if err != nil {
            common.LogDebug(common.Debug, "Write to packet Error:,", err.Error())
            return
        }
    //release memory
    success := newp.SendPacket(1)
    if !success {
        common.LogWarning(common.Debug, "Can not release packet,this may cause mempool full.")
    }

I think there should have way to handle packets generate by myself,not only to send it,but to drop it.

gshimansky commented 4 years ago

To put a packet into a flow you can use SendPacket. It sends the packet from target port and releases memory that it occupied.

cang233 commented 4 years ago

Yeah I used it,but I got an error...

But my operations is jumped out of the graph,now I only operate the single packet and can not release it.If there are any ways to put the packet generated by myself to a flow? Or I can release the single packet memory by myself ? I tried the SendPacket() to release it but meet a error:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7aad97]

runtime stack:
runtime.throw(0xab9c4a, 0x2a)
  /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
  /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 301 [syscall]:
runtime.cgocall(0x7ae2a0, 0xc000066e10, 0x0)
  /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000066de0 sp=0xc000066da8 pc=0x438ffb
github.com/intel-go/nff-go/internal/low._Cfunc_directSend(0x16a633e00, 0x16a630001, 0x0)
  _cgo_gotypes.go:500 +0x4a fp=0xc000066e10 sp=0xc000066de0 pc=0x5b704a
github.com/intel-go/nff-go/internal/low.DirectSend.func1(0x16a633e00, 0x1, 0x16a633f00)
  /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/internal/low/low.go:48 +0x60 fp=0xc000066e48 sp=0xc000066e10 pc=0x5ba150
github.com/intel-go/nff-go/internal/low.DirectSend(0x16a633e00, 0x1, 0x0)
  /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/internal/low/low.go:48 +0x35 fp=0xc000066e70 sp=0xc000066e48 pc=0x5b7ff5
github.com/intel-go/nff-go/packet.(*Packet).SendPacket(...)
  /home/pvs1/gopath/pkg/mod/github.com/intel-go/nff-go@v0.9.1/packet/packet.go:848
flowparser/flows.save2file(0xc000b854a0)
  /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter_save_handler.go:119 +0x235 fp=0xc000066f60 sp=0xc000066e70 pc=0x5c1825
flowparser/flows.saveSamples(0xc000b854a0)
  /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter_save_handler.go:72 +0xe3 fp=0xc000066fb0 sp=0xc000066f60 pc=0x5c15d3
flowparser/flows.init.2.func1.3(0xc0000140d0, 0xc000b854a0)
  /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter.go:49 +0x2b fp=0xc000066fd0 sp=0xc000066fb0 pc=0x5c2afb
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000066fd8 sp=0xc000066fd0 pc=0x490d71
created by flowparser/flows.init.2.func1
  /home/pvs1/gopath/src/flowparser/flows/ipv4_tcp_exporter.go:48 +0xf7

Here is part of my code:

  newp, err := packet.NewPacket()
  if err != nil {
      common.LogDebug(common.Debug, "New packet Error:,", err.Error())
      return
  }
        packet.GeneratePacketFromByte(newp, v.RawBytes)
      if !success {
          common.LogDebug(common.Debug, "Can not generate packet from byte.")
          return
      }
        newp.WritePcapOnePacket(file)
      if err != nil {
          common.LogDebug(common.Debug, "Write to packet Error:,", err.Error())
          return
      }
  //release memory
  success := newp.SendPacket(1)
  if !success {
      common.LogWarning(common.Debug, "Can not release packet,this may cause mempool full.")
  }

I think there should have way to handle packets generate by myself,not only to send it,but to drop it.

aregm commented 4 years ago

I guess what you want to do is to create a mirror stream and dump it into a file. Please take a look at https://github.com/intel-go/nff-go/blob/master/examples/clonablePcapDumper.go for a clonable version, or to https://github.com/intel-go/nff-go/blob/master/examples/dump.go for a simple version.

cang233 commented 4 years ago

@aregm Yeah,that's part of my implementation! Thanks a lot.By the way,I also want to ask that the reason why there's a error when calling SendPacket(),and how should I understand the UserContext structure?It has less doc in Wiki page.