Open justinruggles opened 10 years ago
I don't have a specific test case because this is happening randomly and infrequently on our servers, but I do have a backtrace. https://gist.github.com/justinruggles/cbd8d920976f9c58f6b6
The deadlock is in:
goroutine 23970 [semacquire, 841 minutes]: sync.runtime_Semacquire(0xc209ae282c) /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30 sync.(*Mutex).Lock(0xc209ae2828) /usr/local/go/src/pkg/sync/mutex.go:66 +0xd6 github.com/mikespook/gearman-go/client.(*Client).Status(0xc21360b730, 0xc20f6ebda0, 0x13, 0x0, 0x0, 0x0) /home/vimeo/stack/lib/go/src/github.com/mikespook/gearman-go/client/client.go:261 +0x280 main.(*JobServer).sendGearmanJob(0xc20801a8c0, 0xc20d495440, 0x6f5d50, 0xa, 0x0, 0x0) /home/vimeo/jobserver/client.go:165 +0x380 main.(*JobServer).handlePutJob(0xc20801a8c0, 0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00, 0xc20cef1b00) /home/vimeo/jobserver/server.go:262 +0x707 main.*JobServer.(main.handlePutJob)·fm(0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00, 0x1) /home/vimeo/jobserver/server.go:468 +0x44 vimeo/http.(*SeqHandlerFuncImpl).ServeHTTP(0xc208040080, 0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00, 0xc20804e901) /home/vimeo/stack/lib/go/src/vimeo/http/handler.go:32 +0x4e vimeo/http.(*RestfulHandler).ServeHTTP(0xc20800edb0, 0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00) /home/vimeo/stack/lib/go/src/vimeo/http/handler.go:113 +0x2ac vimeo/http.(*Server).ServeHTTP(0xc20801a910, 0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00) /home/vimeo/stack/lib/go/src/vimeo/http/server.go:34 +0xfc net/http.serverHandler.ServeHTTP(0xc208004360, 0x7fa8ae94ab78, 0xc20d4dfae0, 0xc20ebffa00) /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f net/http.(*conn).serve(0xc20814f500) /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e created by net/http.(*Server).Serve /usr/local/go/src/pkg/net/http/server.go:1721 +0x313
Here is a snippet of my client code:
func (this *JobServer) sendGearmanJob(job *Job, worker string) error { logger := job.logger c, err := client.New(client.Network, "localhost:4730") if err != nil { logger.Println(err) return err } c.ErrorHandler = func(e error) { logger.Println(err) } payload, err := json.Marshal(job) if err != nil { c.Close() return err } jobHandler := func(resp *client.Response) { // response handling code here } handle, err := c.Do(worker, payload, client.JobNormal, jobHandler) if err != nil { return err } st, err := c.Status(handle) if err == nil { logger.Printf("Status: %+v", st) } else { logger.Printf("Status: %v", err) } return nil }
I don't have a specific test case because this is happening randomly and infrequently on our servers, but I do have a backtrace. https://gist.github.com/justinruggles/cbd8d920976f9c58f6b6
The deadlock is in:
Here is a snippet of my client code: