Analyzer is crashing in GKE staging

rporres commented 5 years ago


$ kubectl -n lookout logs -p lookout-terraform-analyzer-5c64458d86-wh4kq
time="2018-11-21T11:40:14.639426568Z" level=info msg="server has started on 'ipv4://0.0.0.0:10303'"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x87c32a]

goroutine 2940 [running]:
github.com/src-d/lookout-terraform-analyzer/vendor/github.com/src-d/lookout/util/grpchelper.newServerRequestLogger(0x0, 0x0, 0x9aa334, 0x1e, 0x18, 0xc42010e9e0)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/github.com/src-d/lookout/util/grpchelper/logger.go:91 +0x27a
github.com/src-d/lookout-terraform-analyzer/vendor/github.com/src-d/lookout/util/grpchelper.UnaryServerInterceptor.func1(0xa01140, 0xc42013d410, 0x977740, 0xc420088580, 0xc42010e9c0, 0xc42010e9e0, 0x8fe2a0, 0xcf56e0, 0x986440, 0xc420460200)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/github.com/src-d/lookout/util/grpchelper/logger.go:27 +0xa6
github.com/src-d/lookout-terraform-analyzer/vendor/gopkg.in/src-d/lookout-sdk.v0/pb._Analyzer_NotifyReviewEvent_Handler(0x938dc0, 0xc42010e8c0, 0xa01140, 0xc42013d410, 0xc4201da150, 0xc42010e880, 0x0, 0x0, 0xc42016a300, 0xb5)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/gopkg.in/src-d/lookout-sdk.v0/pb/service_analyzer.pb.go:185 +0x167
github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420001200, 0xa03660, 0xc420001500, 0xc420460200, 0xc42013c9f0, 0xcc6c20, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc/server.go:982 +0x4f9
github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420001200, 0xa03660, 0xc420001500, 0xc420460200, 0x0)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc/server.go:1208 +0x1318
github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc420170260, 0xc420001200, 0xa03660, 0xc420001500, 0xc420460200)
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    /home/travis/gopath/src/github.com/src-d/lookout-terraform-analyzer/vendor/google.golang.org/grpc/server.go:684 +0xa1

meyskens commented 5 years ago

Interesting that the trace only talks about gRPC

meyskens commented 5 years ago

Potential fix: https://github.com/src-d/lookout-terraform-analyzer/pull/8 If that doesn't help it might be a case for @src-d/applications since this happens in github.com/src-d/lookout/util/grpchelper/logger.go

smacker commented 5 years ago

You need to do

log.DefaultFactory = &log.LoggerFactory{Level: conf.LogLevel}
log.DefaultLogger = log.New(nil)

in your main package.

Sdk update won't help because you use helpers from lookout server, not lookouk-sdk. Lookout server isn't supposed to be used as a library.

meyskens commented 5 years ago

@smacker thank you! didn't spot that! I looked at the gometalint one and found those helpers, didn't know they were not supposed to be used

smacker commented 5 years ago

um. gometalint was created before sdk started to exist 😅 if you miss something in sdk feel free to create issues/feature requests. We need some feedback :)

meyskens commented 5 years ago

@dpordomingo is going to take a look at this one, marking it as blocked for the infra team.

dpordomingo commented 5 years ago

:point_up: @meyskens done by #9

rporres commented 5 years ago

https://github.com/src-d/lookout-terraform-analyzer/releases/tag/v0.1.0 was released and deployed successfully fixing the crashes. It also contained changes from #10

src-d / lookout-terraform-analyzer

Analyzer is crashing in GKE staging #7