search

kakao / MRTE-Collector

MRTE(MySQL Realtime Traffic Emulator) Collector
19 stars 9 forks source link

panic: Memory usage is too high (80576512 > 67108864), Increase memory limit or need to decrease memory usage #1

Open allenmaestro opened 8 years ago

allenmaestro commented 8 years ago

On mysql server side, I have 64GB memory and 46GB momory is being used. Whenever I execute collector, it shows the following message. Is there any option to decrease memory usage? or Do I have to test this module on the low memory usage servers?

[INFO] MRTECollector : device: ' bond0 ', link_type:' DLT_EN10MB ( 1 )', capture_filter:' tcp dst port 3306 ' [INFO] Send init database information of all session (1). Wait ... [INFO] Send init database information of all session (2). Done

DateTime TotalPacket ValidPacket PacketDropped PacketIfDropped WaitingQueueCnt MQError 2016-02-29 10:30:47 3653 3056 99 0 3 0 General statistics. Alloc : 884024 // bytes allocated and still in use TotalAlloc : 56703072 // bytes allocated (even if freed) Sys : 10918136 // bytes obtained from system (sum of XxxSys below) Lookups : 12 // number of pointer lookups Mallocs : 1349668 // number of mallocs Frees : 1333934 // number of frees

Main allocation heap statistics. HeapAlloc : 884024 // bytes allocated and still in use HeapSys : 6651904 // bytes obtained from system HeapIdle : 4956160 // bytes in idle spans HeapInuse : 1695744 // bytes in non-idle span HeapReleased : 0 // bytes released to the OS HeapObjects : 15734 // total number of allocated objects

Low-level fixed-size structure allocator statistics. Inuse is bytes used now. Sys is bytes obtained from system. StackInuse : 688128 // bytes used by stack allocator StackSys : 688128 MSpanInuse : 25424 // mspan structures MSpanSys : 81920 MCacheInuse : 7248 // mcache structures MCacheSys : 32768 BuckHashSys : 1451661 // profiling bucket hash table GCSys : 233984 // GC metadata OtherSys : 1777771 // other system allocations

Garbage collector statistics. NextGC : 4194304 // next collection will happen when HeapAlloc ≥ this amount LastGC : 1456709447293828743 // end time of last collection (nanoseconds since 1970) PauseTotalNs : 20149122 NumGC : 14 EnableGC : true DebugGC : false panic: Memory usage is too high (80576512 > 67108864), Increase memory limit or need to decrease memory usage

goroutine 27 [running]: main.checkMemoryUsage(0x4000000) /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:445 +0x2dd main.main.func1(0xc8200e0028, 0x5) /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:319 +0x9bc created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:342 +0x1c81

goroutine 1 [syscall, locked to thread]: _/opt/mrte/MRTE-Collector-master/src/github.com/miekg/pcap._Cfunc_hack_pcap_nextex(0x7f3db4000b20, 0xc82017e2c0, 0xc82017e2c8, 0x7f3d00000000) ??:0 +0x39 /opt/mrte/MRTE-Collector-master/src/github.com/miekg/pcap.(*Pcap).NextEx(0xc8200e0028, 0x0, 0xc820055c60) /opt/mrte/MRTE-Collector-master/src/github.com/miekg/pcap/pcap.go:187 +0x96 main.main() /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:351 +0x1ff1

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]: os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1 /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 18 [IO wait]: net.runtime_pollWait(0x7f3dc012a6c0, 0x72, 0xc82000e1c0) /usr/lib/golang/src/runtime/netpoll.go:157 +0x60 net.(_pollDesc).Wait(0xc82007c0d0, 0x72, 0x0, 0x0) /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x3a net.(_pollDesc).WaitRead(0xc82007c0d0, 0x0, 0x0) /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x36 net.(_netFD).Read(0xc82007c070, 0xc8200e5000, 0x1000, 0x1000, 0x0, 0x7f3dc0125050, 0xc82000e1c0) /usr/lib/golang/src/net/fd_unix.go:232 +0x23a net.(_conn).Read(0xc8200e0000, 0xc8200e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:172 +0xe4 bufio.(_Reader).fill(0xc8200e6180) /usr/lib/golang/src/bufio/bufio.go:97 +0x1e9 bufio.(Reader).Read(0xc8200e6180, 0xc8203569a0, 0x7, 0x7, 0x1, 0x0, 0x0) /usr/lib/golang/src/bufio/bufio.go:207 +0x260 io.ReadAtLeast(0x7f3dbc6cb120, 0xc8200e6180, 0xc8203569a0, 0x7, 0x7, 0x7, 0x0, 0x0, 0x0) /usr/lib/golang/src/io/io.go:298 +0xe6 io.ReadFull(0x7f3dbc6cb120, 0xc8200e6180, 0xc8203569a0, 0x7, 0x7, 0x1, 0x0, 0x0) /usr/lib/golang/src/io/io.go:316 +0x62 /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(reader).ReadFrame(0xc820075ee0, 0x0, 0x0, 0x0, 0x0) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/read.go:49 +0xbd /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(Connection).reader(0xc8200ea000, 0x7f3dbc6cb088, 0xc8200e0000) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/connection.go:464 +0x172 created by /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.Open /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/connection.go:220 +0x3e3

goroutine 19 [select]: _/opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(Connection).heartbeater(0xc8200ea000, 0x2540be400, 0xc8200e61e0) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/connection.go:494 +0x505 created by /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(_Connection).openTune /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/connection.go:713 +0x805

goroutine 20 [runnable]: sync.(Mutex).Unlock(0xc8200ea00c) /usr/lib/golang/src/sync/mutex.go:99 /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(Connection).send(0xc8200ea000, 0x7f3dbc6cb1f0, 0xc820285880, 0x0, 0x0) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/connection.go:320 +0xa5 /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(Channel).sendOpen(0xc820136300, 0x7f3da985c0d0, 0xc820403560, 0x0, 0x0) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/channel.go:213 +0x2ce /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp.(Channel).Publish(0xc820136300, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc820360000, 0xc8201afbc0, 0x730ba0, 0xa, 0x7219b0, ...) /opt/mrte/MRTE-Collector-master/src/github.com/streadway/amqp/channel.go:1306 +0x42a /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.(*MysqlRequest).publish(0xc82029fa70, 0xc820136300, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed7c, 0x4, 0x4, 0x1, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:82 +0xc81 _/opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.PublishMessage(0x0, 0x1, 0xc8200ea000, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed70, 0x10, 0x10, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:31 +0x14a created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:243 +0x1b9d

goroutine 21 [chan receive]: _/opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.PublishMessage(0x1, 0x1, 0xc8200ea000, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed70, 0x10, 0x10, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:30 +0xb6 created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:243 +0x1b9d

goroutine 22 [chan receive]: _/opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.PublishMessage(0x2, 0x1, 0xc8200ea000, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed70, 0x10, 0x10, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:30 +0xb6 created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:243 +0x1b9d

goroutine 23 [chan receive]: _/opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.PublishMessage(0x3, 0x1, 0xc8200ea000, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed70, 0x10, 0x10, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:30 +0xb6 created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:243 +0x1b9d

goroutine 24 [chan receive]: _/opt/mrte/MRTE-Collector-master/src/kakao.com/mrte.PublishMessage(0x4, 0x1, 0xc8200ea000, 0x7fff24d09615, 0x4, 0x7fff24d0961c, 0x0, 0xc82000ed70, 0x10, 0x10, ...) /opt/mrte/MRTE-Collector-master/src/kakao.com/mrte/RabbitMQPublisher.go:30 +0xb6 created by main.main /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:243 +0x1b9d

goroutine 25 [select, locked to thread]: runtime.gopark(0x7a7f50, 0xc820042f28, 0x724f48, 0x6, 0x18, 0x2) /usr/lib/golang/src/runtime/proc.go:185 +0x163 runtime.selectgoImpl(0xc820042f28, 0x0, 0x18) /usr/lib/golang/src/runtime/select.go:392 +0xa64 runtime.selectgo(0xc820042f28) /usr/lib/golang/src/runtime/select.go:212 +0x12 runtime.ensureSigM.func1() /usr/lib/golang/src/runtime/signal1_unix.go:227 +0x353 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

goroutine 26 [chan receive]: main.addSignalHandler.func1(0xc8200e64e0, 0x7fff24d0963f, 0xc, 0xcea, 0x7fff24d09659, 0xb, 0x7fff24d09676, 0xc, 0xc8200e0028) /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:520 +0x87 created by main.addSignalHandler /opt/mrte/MRTE-Collector-master/src/MRTECollector.go:532 +0x579

allenmaestro commented 8 years ago

I've figured out by increasing --max_mem_mb value.