m-lab / etl-gardener

Gardener provides services for maintaining and reprocessing mlab data.
Apache License 2.0
13 stars 5 forks source link

Fix NextJob panic #411

Closed stephen-soltesz closed 1 year ago

stephen-soltesz commented 1 year ago

This change fixes a "panic on nil reference" in the job tracker handler.

The JobService can a return nil value when there are no jobs available to process (such as no historical data or daily only datatypes). The Tracker handler for NextJob failed to check for the nil case, which resulted in non-fatal panics. While the parser was resilient to these failures, the gardener should handle this case correctly.

Now, a client will receive an HTTP "Internal Server Error" for Next job requests that return nil. The parser remains resilient to this case.

2022/10/05 05:42:59 job-service.go:71: 20190110:ndt/ndt7 has no files archive-measurement-lab
2022/10/05 05:42:59 server.go:3197: http: panic serving 10.12.7.2:56470: runtime error: invalid memory address or nil pointer dereference
goroutine 164329685 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1825 +0xbf
panic({0xc0b580, 0x1361540})
        /usr/local/go/src/runtime/panic.go:844 +0x258
github.com/m-lab/etl-gardener/tracker.(*Handler).nextJob(0xc0000b5a28, {0xe613d0, 0xc000bc40e0}, 0xc000580600?)
        /go/src/github.com/m-lab/etl-gardener/tracker/handler.go:118 +0xc0
github.com/m-lab/etl-gardener/tracker.(*Handler).nextJobV2(0x0?, {0xe613d0, 0xc000bc40e0}, 0x4f8da9?)
        /go/src/github.com/m-lab/etl-gardener/tracker/handler.go:137 +0x3b
net/http.HandlerFunc.ServeHTTP(0x7fbc8ba37b18?, {0xe613d0?, 0xc000bc40e0?}, 0x40d5c5?)
        /usr/local/go/src/net/http/server.go:2084 +0x2f
net/http.(*ServeMux).ServeHTTP(0x0?, {0xe613d0, 0xc000bc40e0}, 0xc000616400)
        /usr/local/go/src/net/http/server.go:2462 +0x149
net/http.serverHandler.ServeHTTP({0xc0005034d0?}, {0xe613d0, 0xc000bc40e0}, 0xc000616400)
        /usr/local/go/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0004b86e0, {0xe61d98, 0xc0004c5b60})
        /usr/local/go/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3071 +0x4db

This change is Reviewable

coveralls commented 1 year ago

Pull Request Test Coverage Report for Build 3580


Totals Coverage Status
Change from base Build 3577: 0.2%
Covered Lines: 1235
Relevant Lines: 1590

💛 - Coveralls