Open pmyjavec opened 7 years ago
@nathanielc,
My bad, I think I got it wrong yesterday.
According to @kostasb the behaviour is pretty normal and is triggered when the process runs out of memory and generates a crash log does that sound accurate?
If this is the case, do you think it's possible to surround the crash log with something to indicate that's what we're looking at? It would be really neat to have something like:
2017/05/15 18:35:19 --- BEGIN CRASH LOG ---
goroutine 2323 [chan receive, 10 minutes]:
github.com/influxdata/kapacitor.(*group).run(0xc4207d4050, 0x0, 0x0)
/home/builder/go/src/github.com/influxdata/kapacitor/join.go:324 +0xb2
github.com/influxdata/kapacitor.(*JoinNode).getGroup.func1(0xc4207d4050, 0xc4204e6160, 0xc4204e3020)
/home/builder/go/src/github.com/influxdata/kapacitor/join.go:278 +0x35
created by github.com/influxdata/kapacitor.(*JoinNode).getGroup
/home/builder/go/src/github.com/influxdata/kapacitor/join.go:286 +0x2c5
net.runtime_pollWait(0x7f770b3205b0, 0x72, 0xa)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc479db5db0, 0x72, 0xc455e1e7b0, 0xc420016140)
net.(*pollDesc).waitRead(0xc479db5db0, 0x12116c0, 0xc420016140)
net.(*netFD).Read(0xc479db5d50, 0xc467381000, 0x1000, 0x1000, 0x0, 0x12116c0, 0xc420016140)
net.(*conn).Read(0xc42036c5f8, 0xc467381000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
2017/05/15 18:35:19 --- END CRASH LOG ---
Thanks!
@pmyjavec It would be nice to surround it but its part of the Go runtime so I don't think we have access to how it outputs.
The fact that its crashing is not good, can you share the very first lines of a crash, sometimes the crash is an OOMing error, other times its a bug in the code, so it would be good to confirm its not the later.
@nathanielc please see the issue referenced by @kostasb for details on the crashes
@nathanielc,
Actually I managed to find the library used that can wrap the panic in something informative. Here it is: https://github.com/mitchellh/panicwrap
Using that would be really cool, what do you think? By the way I would be interested in helping implement something like that :)
Good find I wasn't able to find anything like that in my quick search. Unfortunately I do not think it is a viable solution. The panicwrap is a process wrapper not simply a library. So that would mean that to run Kapacitor you would have to run two process, the panicwrap process and the actual Kapacitor daemon process. That change has significant repercussions for most deployment mechanisms.
Hey @nathanielc, I would like to understand how this would have significant repercussions for most deployment mechanisms? As a user / SRE working with Kapacitor I can't see how this would be a negative change or force me to change my deployment strategy in anyway.
Users wouldn't actually need to run two commands, they would still just use kapacitord
as the entry point and panic wrapper
just re-executes the program.
Sure this would run the main()
loop as a sub-process / fork, but that's it's purpose and is really quite transparent to the user while adding major benefits such as useful control over the crash logging and provide a more helpful user experience.
While running v1.2.2 Kapacitor seems to be dumping a bunch of traces to standard out.
Here are some of the example traces:
Here is a link to the running configuration: https://gist.github.com/pmyjavec/57910435a55ef4ba11e7982b40e86c66