search

metal-stack / metal-api

API to manage and control plane resources like machines, switches, operating system images, machine sizes, networks, IP addresses and more
GNU Affero General Public License v3.0
132 stars 9 forks source link

hanging go routines after metal-db restart #270

Open mwennrich opened 2 years ago

mwennrich commented 2 years ago

image

$ kubectl get pods -A|grep metal-d
metal-control-plane                            metal-db-0                                                2/2     Running     0          4h43m

$ kubectl get pods -A|grep metal-api
metal-control-plane                            metal-api-69c7c48557-6v4r8                                1/1     Running     0          4d12h
metal-control-plane                            metal-api-69c7c48557-8vscq                                1/1     Running     0          4d12h
metal-control-plane                            metal-api-69c7c48557-dws7h                                1/1     Running     0          4d14h
metal-control-plane                            metal-api-69c7c48557-khbd4                                1/1     Running     0          4h40m
metal-control-plane                            metal-api-69c7c48557-lrmwg                                1/1     Running     0          4h40m
mwennrich commented 2 years ago 
goroutine profile: total 1553
669 @ 0x439ad6 0x432153 0x4655e9 0x4da5b2 0x4db91a 0x4db908 0x5c24e9 0x5d2e45 0x498a3a 0xd4f913 0xd4f8e4 0xd4ba17 0xd4ac25 0x46ad41
#   0x4655e8    internal/poll.runtime_pollWait+0x88                 /usr/local/go/src/runtime/netpoll.go:234
#   0x4da5b1    internal/poll.(*pollDesc).wait+0x31                 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#   0x4db919    internal/poll.(*pollDesc).waitRead+0x259                /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#   0x4db907    internal/poll.(*FD).Read+0x247                      /usr/local/go/src/internal/poll/fd_unix.go:167
#   0x5c24e8    net.(*netFD).Read+0x28                          /usr/local/go/src/net/fd_posix.go:56
#   0x5d2e44    net.(*conn).Read+0x44                           /usr/local/go/src/net/net.go:183
#   0x498a39    io.ReadAtLeast+0x99                         /usr/local/go/src/io/io.go:328
#   0xd4f912    io.ReadFull+0x52                            /usr/local/go/src/io/io.go:347
#   0xd4f8e3    gopkg.in/rethinkdb/rethinkdb-go%2ev6.(*Connection).read+0x23        /go/pkg/mod/gopkg.in/rethinkdb/rethinkdb-go.v6@v6.2.1/connection_helper.go:16
#   0xd4ba16    gopkg.in/rethinkdb/rethinkdb-go%2ev6.(*Connection).readResponse+0x56    /go/pkg/mod/gopkg.in/rethinkdb/rethinkdb-go.v6@v6.2.1/connection.go:405
#   0xd4ac24    gopkg.in/rethinkdb/rethinkdb-go%2ev6.(*Connection).readSocket+0x24  /go/pkg/mod/gopkg.in/rethinkdb/rethinkdb-go.v6@v6.2.1/connection.go:258

669 @ 0x439ad6 0x449872 0xd4ae33 0x46ad41
#   0xd4ae32    gopkg.in/rethinkdb/rethinkdb-go%2ev6.(*Connection).processResponses+0x192   /go/pkg/mod/gopkg.in/rethinkdb/rethinkdb-go.v6@v6.2.1/connection.go:282
Gerrit91 commented 2 years ago

I am unable to reproduce this. Could be the #271 helps in this scenario, but don't know.