search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
102 stars 4.92k forks source link

Crash parsing Redis traffic #384

Closed hiteshmahajan closed 8 years ago

hiteshmahajan commented 8 years ago

Too many logs getting generated inside /var/log/messages

Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:111: 
Stacktrace:
/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/logp/log.go:111 
(0x48bfb6)#012/usr/local/go/src/runtime/asm_amd64.s:437 
(0x47d72e)#012/usr/local/go/src/runtime/panic.go:423 
(0x44d329)#012/usr/local/go/src/runtime/panic.go:42 
(0x44b9e9)#012/usr/local/go/src/runtime/sigpanic_unix.go:24 
(0x46228a)#012/go/src/github.com/elastic/packetbeat/protos/redis/redis.go:579
(0x51af62)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:87 
(0x5202f3)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:173 
(0x52142d)#012/go/src/github.com/elastic/packetbeat/decoder/decoder.go:136 
(0x6c6041)#012/go/src/github.com/elastic/packetbeat/sniffer/sniffer.go:352 
(0x532a09)#012/go/src/github.com/elastic/packetbeat/packetbeat.go:212 
(0x422f2b)#012/usr/local/go/src/runtime/asm_amd64.s:1696 
(0x47fa71)
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:110: ParseRedis exception. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:111: 
Stacktrace:
/go/src/github.com/elastic/packetbeat/Godeps/_workspace/src/github.com/elastic/libbeat/logp/log.go:111 
(0x48bfb6)#012/usr/local/go/src/runtime/asm_amd64.s:437 
(0x47d72e)#012/usr/local/go/src/runtime/panic.go:423 
(0x44d329)#012/usr/local/go/src/runtime/panic.go:42 
(0x44b9e9)#012/usr/local/go/src/runtime/sigpanic_unix.go:24 
(0x46228a)#012/go/src/github.com/elastic/packetbeat/protos/redis/redis.go:579 
(0x51af62)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:87 
(0x5202f3)#012/go/src/github.com/elastic/packetbeat/protos/tcp/tcp.go:173 
(0x52142d)#012/go/src/github.com/elastic/packetbeat/decoder/decoder.go:136 
(0x6c6041)#012/go/src/github.com/elastic/packetbeat/sniffer/sniffer.go:352 
(0x532a09)#012/go/src/github.com/elastic/packetbeat/packetbeat.go:212 
(0x422f2b)#012/usr/local/go/src/runtime/asm_amd64.s:1696 
(0x47fa71)
Nov 18 12:51:47 ip-10-168-0-218 /usr/bin/packetbeat[27009]: log.go:110: ParseRedis exception. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
ruflin commented 8 years ago

Based on your log files I assume you are sniffing Redis traffic with Packetbeat. Which version of packetbeat are you using? Would it be possible for you to send us some example pcap messages that cause this issue?

hiteshmahajan commented 8 years ago

Yes we are using redis to store data

output:
  ### Redis as output
  redis:
    enabled: true
    host: x.x.x.x
    port: 6379

packetbeat version 1.0.0-rc1 (amd64)

$5
RPUSH
$10
packetbeat
$490
{"@timestamp":"2015-11-18T09:42:51.282Z","bytes_in":1270,"bytes_out":917,"client_ip":"10.148.128.180","client_port":58972,"client_proc":"","client_server":"","count":1,"direction":"out","http":{"code":200,"content_length":214,"phrase":"OK"},"ip":"10.168.0.218","method":"POST","params":"","path":"/notification/notifyme","port":80,"proc":"","query":"POST /notification/notifyme","responsetime":25,"server":"","shipper
15:12:51.555048 IP 10.168.0.218.55037 > 10.168.231.212.6379: tcp 75
E....8@.@..B
...
..........4..#C...s.o.....
.x..-.".":"...warpspeed...","status":"OK","tags":["warp-frontend"],"type":"http"}

15:12:51.594480 IP 10.168.231.212.6379 > 10.168.0.218.55037: tcp 0
@.9.@.
...
.........#C.........5.....
-.#..x..
15:12:51.939459 IP 10.168.231.212.6379 > 10.168.0.218.55037: tcp 1083
E..o..@.9.<}
...
.........#C...............
-.$[.x..-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'
-ERR command not allowed when used memory > 'maxmemory'

15:12:51.939480 IP 10.168.0.218.55037 > 10.168.231.212.6379: tcp 0
E..4.9@.@...
...
.............'~...........
.x.e-.$[
15:12:51.939725 IP 10.168.0.218.55044 > 10.168.231.212.6379: tcp 0
E..<.K@.@..r
...
.........}.......9............
.x.e........
andrewkroh commented 8 years ago

It looks like it is crashing here. Would it be possible to run packetbeat with the -dump some/dir/captured-data.pcap command line option and send me the capture?

I will then replay the capture file through packetbeat to recreate crash, develop a test case and a fix. I run something like this to replay the file through: packetbeat -c packetbeat.yml -e -v -d "redis" -t -I some/dir/captured-data.pcap -waitstop 30

jalberto commented 8 years ago

I have same error in rc2, @andrewkroh do you want one pcap from my system too?

andrewkroh commented 8 years ago

@jalberto Yes, that would be great because I still need a pcap for this issue. Thanks

jalberto commented 8 years ago

@andrewkroh check your email :)

voelzmo commented 8 years ago

Can we rename this to have 'redis' in the issue title? It is somehow difficult to find using github's issue search right now. Thanks.

tsg commented 8 years ago

@voelzmo good point, I changed it.

jalberto commented 8 years ago

@andrewkroh @tsg Thanks!

Has this been released?

ruflin commented 8 years ago

@jalberto This has not been release yet as it was just merged an hour ago. If you like I can trigger the nightly builds so you will have a binary to test? Alternative you can build it yourself directly from the source code.

jalberto commented 8 years ago

thanks @ruflin,

I saw a new version on the elastic page, so I was wondering if this was included there, but now I can see the 1.1.0 tag

I can wait, but if you trigger the nightly build I will test it

ruflin commented 8 years ago

@jalberto Nightly builds triggered. It normally takes between 10 and 60 minutes to build them (depending on travis load). They will show up with the most recent timestamp soonish here: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

jalberto commented 8 years ago

Thanks!

On Thu, 26 Nov 2015 at 12:01 Nicolas Ruflin notifications@github.com wrote:

@jalberto https://github.com/jalberto Nightly builds triggered. It normally takes between 10 and 60 minutes to build them (depending on travis load). They will show up with the most recent timestamp soonish here: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

— Reply to this email directly or view it on GitHub https://github.com/elastic/packetbeat/issues/384#issuecomment-159895755.