Deadlock in go-elasticsearch/estransport/connection.go ？！

[bjkxt]

I have some golang programs using go-elasticsearch to access my ES cluster. I found that one of them can't work any more so I use the pprof by http://x.x.x.x/debug/pprof/goroutine?debug=1 to get the goroutines' status. It seems that there was a deadlock between function OnSuccess() (in file estransport/connection.go:138) and function scheduleResurrect() (in file estransport/connection.go:259). What can I do to prevent this situation? Thanks a lot for your advice.

---

some of the pprof's returns are below：

63695 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa9c022 0xa9c02b 0x8cbb55 0x9e8742 0x117a268 0x12921a2 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa9c021 sync.(Mutex).Lock+0xaf1 /data/go/src/sync/mutex.go:81 # 0xa9c02a github.com/elastic/go-elasticsearch/v6/estransport.(*Client).Perform+0xafa /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:234 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

31164 @ 0x439390 0x449513 0x7786e5 0x773feb 0x75a785 0xa9b8b5 0x8cbb55 0x9e8742 0x117a268 0x12921a2 0x1294b47 0x469661 # 0x7786e4 net/http.(Transport).getConn+0x584 /data/go/src/net/http/transport.go:1350 # 0x773fea net/http.(Transport).roundTrip+0x76a /data/go/src/net/http/transport.go:569 # 0x75a784 net/http.(Transport).RoundTrip+0x34 /data/go/src/net/http/roundtrip.go:17 # 0xa9b8b4 github.com/elastic/go-elasticsearch/v6/estransport.(Client).Perform+0x384 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:258 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

5749 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xaa0364 0xaa01d6 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xaa0363 sync.(Mutex).Lock+0x1d3 /data/go/src/sync/mutex.go:81 # 0xaa01d5 github.com/elastic/go-elasticsearch/v6/estransport.(*statusConnectionPool).scheduleResurrect.func1+0x45 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/connection.go:259

5262 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa9bcc0 0xa9bcc9 0x8cbb55 0x9e8742 0x117a268 0x12921a2 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa9bcbf sync.(Mutex).Lock+0x78f /data/go/src/sync/mutex.go:81 # 0xa9bcc8 github.com/elastic/go-elasticsearch/v6/estransport.(*Client).Perform+0x798 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:295 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

25 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa9c022 0xa9c02b 0x8cbb55 0x9e8742 0x1177d3e 0x129211d 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa9c021 sync.(Mutex).Lock+0xaf1 /data/go/src/sync/mutex.go:81 # 0xa9c02a github.com/elastic/go-elasticsearch/v6/estransport.(*Client).Perform+0xafa /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:234 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

6 @ 0x439390 0x449513 0x7786e5 0x773feb 0x75a785 0xa9b8b5 0x8cbb55 0x9e8742 0x1177d3e 0x129211d 0x1294b47 0x469661 # 0x7786e4 net/http.(Transport).getConn+0x584 /data/go/src/net/http/transport.go:1350 # 0x773fea net/http.(Transport).roundTrip+0x76a /data/go/src/net/http/transport.go:569 # 0x75a784 net/http.(Transport).RoundTrip+0x34 /data/go/src/net/http/roundtrip.go:17 # 0xa9b8b4 github.com/elastic/go-elasticsearch/v6/estransport.(Client).Perform+0x384 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:258 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

1 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa977dd 0xa9775d 0xa9bc81 0x8cbb55 0x9e8742 0x117a268 0x12921a2 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa977dc sync.(Mutex).Lock+0x11c /data/go/src/sync/mutex.go:81 # 0xa9775c github.com/elastic/go-elasticsearch/v6/estransport.(statusConnectionPool).OnSuccess+0x9c /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/connection.go:138 # 0xa9bc80 github.com/elastic/go-elasticsearch/v6/estransport.(Client).Perform+0x750 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:296 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

1 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa9bbfb 0xa9bc04 0x8cbb55 0x9e8742 0x117a268 0x12921a2 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa9bbfa sync.(Mutex).Lock+0x6ca /data/go/src/sync/mutex.go:81 # 0xa9bc03 github.com/elastic/go-elasticsearch/v6/estransport.(*Client).Perform+0x6d3 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:278 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

1 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xa9bcc0 0xa9bcc9 0x8cbb55 0x9e8742 0x1177d3e 0x129211d 0x1294b47 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xa9bcbf sync.(Mutex).Lock+0x78f /data/go/src/sync/mutex.go:81 # 0xa9bcc8 github.com/elastic/go-elasticsearch/v6/estransport.(*Client).Perform+0x798 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/estransport.go:295 # 0x8cbb54 github.com/elastic/go-elasticsearch/v6/esapi.SearchRequest.Do+0x1244 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:335 # 0x9e8741 github.com/elastic/go-elasticsearch/v6/esapi.newSearchFunc.func1+0x141 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/esapi/api.search.go:25

1 @ 0x439390 0x44a450 0x44a43b 0x44a1b7 0x48380c 0xaa0345 0xaa0204 0x469661 # 0x44a1b6 sync.runtime_SemacquireMutex+0x46 /data/go/src/runtime/sema.go:71 # 0x48380b sync.(Mutex).lockSlow+0xfb /data/go/src/sync/mutex.go:138 # 0xaa0344 sync.(Mutex).Lock+0x1b4 /data/go/src/sync/mutex.go:81 # 0xaa0203 github.com/elastic/go-elasticsearch/v6/estransport.(*statusConnectionPool).scheduleResurrect.func1+0x73 /data/GoProject/pkg/mod/github.com/elastic/go-elasticsearch/v6@v6.8.10/estransport/connection.go:262

[skyzhp]

I have the same problem , how do you resolve it?

[AkingZhang0607]

@skyzhp @bjkxt Maybe you need to read the body of response, and invoke resp.Body.Close().

[xiaobai5150]

I have the same problem , have solution to resolve it ?

[thomas-long-f3]

Also seeing this issue in the 7.x branch.