search

lf-edge / eve

EVE is Edge Virtualization Engine
https://www.lfedge.org/projects/eve/
Apache License 2.0
473 stars 159 forks source link

stack traces should be a single message #1356

Open deitch opened 4 years ago

deitch commented 4 years ago

Continues from this eden issue.

When we get a panic, a stack trace appears in pillar.err, which is as planned. However, that stack trace appears on multiple lines, causing it to be multiple LogEntry, which spreads it throughout the log file and makes it hard to work with.

Here is an example:

2020-08-25T10:21:50.992887+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;panic: runtime error: index out of range
2020-08-25T10:21:51.001122+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;
2020-08-25T10:21:51.030452+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;goroutine 1 [running]:
2020-08-25T10:21:51.037369+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.doUpdateContentTree(0xc00046ca80, 0xc0001e03
2020-08-25T10:21:51.047754+00:00 linuxkit-525400123456 pillar.out {"file":"/pillar/cmd/zedagent/parseconfig.go:1451","func":"github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseUnderlayNetwo
2020-08-25T10:21:51.056504+00:00 linuxkit-525400123456 pillar.out {"file":"/pillar/cmd/zedagent/parseconfig.go:520","func":"github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseAppInstanceCon
2020-08-25T10:21:51.092566+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;      /pillar/cmd/volumemgr/updatestatus.go:220 +0x365c
2020-08-25T10:21:51.104269+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.updateContentTree(0xc00046ca80, 0xc0001e03c0
2020-08-25T10:21:51.110450+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;      /pillar/cmd/volumemgr/handlecontent.go:257 +0xb2
2020-08-25T10:21:51.115982+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.handleContentTreeCreateAppImg(0x16a5520, 0xc
2020-08-25T10:21:51.121967+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;      /pillar/cmd/volumemgr/handlecontent.go:21 +0x1f5
2020-08-25T10:21:51.131296+00:00 linuxkit-525400123456 pillar.out {"file":"/pillar/cmd/zedagent/handlecipherconfig.go:81","func":"github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseCipherBl
2020-08-25T10:21:51.137568+00:00 linuxkit-525400123456 pillar.out {"file":"/pillar/cmd/zedagent/handlecipherconfig.go:83","func":"github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseCipherBl
2020-08-25T10:21:51.144265+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;github.com/lf-edge/eve/pkg/pillar/pubsub.handleModify(0x17f2c60, 0xc0004ec380, 0xc0002bb260,
2020-08-25T10:21:51.152633+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;      /pillar/pubsub/subscribe.go:227 +0x6d5
2020-08-25T10:21:51.190357+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;github.com/lf-edge/eve/pkg/pillar/pubsub.(*SubscriptionImpl).ProcessChange(0xc0004ec380, 0x3
2020-08-25T10:21:51.198410+00:00 linuxkit-525400123456 pillar.err 2020-08-25T10:21:50Z,pillar.err;      /pillar/pubsub/subscribe.go:105 +0x31c

The stack trace can be multiple lines, but should be a single LogEntry, so we can work with it properly in logs.

eriknordmark commented 3 years ago

@deitch The size of a logentry is limited; can not exceed the configured size in the controller of a protobuf message which is 32k. Thus we break them up. However, they look more broken up then they should be. Don't know if the CRLF in the string causes issues in the text file; the json output doesn't look like that at all.

rene commented 1 year ago

@rouming , considering kexec features that we have now, is this issue still relevant?